blob: 5c1e96f537fd9158f480acba7a4bb1af63f80383 [file] [log] [blame]
lh9ed821d2023-04-07 01:36:19 -07001 The test suite's file format is very simple and extensible, closely
2resembling XML. All data for a single test case resides in a single
3ASCII file. Labels mark the beginning and the end of all sections, and each
4label must be written in its own line. Comments are either XML-style
5(enclosed with <!-- and -->) or C-style (beginning with #) and must appear
6on their own lines and not alongside actual test data. Most test data files
7are syntactically valid XML, although a few files are not (lack of
8support for character entities and the preservation of CR/LF characters at
9the end of lines are the biggest differences).
10
11 The file begins with a 'testcase' tag, which encompasses the remainder of
12the file.
13
14<testcase>
15
16 Each file is split up in three main sections: reply, client and verify. The
17reply section is used for the server to know what to send as a reply for the
18requests curl sends, the client section defines how the client should behave
19while the verify section defines how to verify that the data stored after a
20command has been run ended up correctly.
21
22 Each main section has a number of available subsections that can be
23specified, that will be checked/used if specified. This document includes all
24the subsections currently supported.
25
26Main sections are 'info', 'reply', 'client' and 'verify'.
27
28<info>
29<keywords>
30A newline-separated list of keywords describing what this test case uses and
31tests. Try to use an already used keyword. These keywords will be used for
32statistical/informational purposes and for choosing or skipping classes
33of tests. "Keywords" must begin with an alphabetic character, "-", "["
34or "{" and may actually consist of multiple words separated by spaces
35which are treated together as a single identifier.
36</keywords>
37</info>
38
39<reply>
40<data [nocheck="yes"] [sendzero="yes"] [base64="yes"]>
41data to be sent to the client on its request and later verified that it arrived
42safely. Set nocheck="yes" to prevent the test script from verifying the arrival
43of this data.
44
45If the data contains 'swsclose' anywhere within the start and end tag, and
46this is a HTTP test, then the connection will be closed by the server after
47this response is sent. If not, the connection will be kept persistent.
48
49If the data contains 'swsbounce' anywhere within the start and end tag, the
50HTTP server will detect if this is a second request using the same test and
51part number and will then increase the part number with one. This is useful
52for auth tests and similar.
53
54'sendzero' set to yes means that the (FTP) server will "send" the data even if
55the size is zero bytes. Used to verify curl's behaviour on zero bytes
56transfers.
57
58'base64' set to yes means that the data provided in the test-file is a chunk
59of data encoded with base64. It is the only way a test case can contain binary
60data. (This attribute can in fact be used on any section, but it doesn't make
61much sense for other sections than "data").
62
63For FTP file listings, the <data> section will be used *only* if you make sure
64that there has been a CWD done first to a directory named 'test-[num]' where
65[num] is the test case number. Otherwise the ftp server can't know from which
66test file to load the list content.
67
68</data>
69<dataNUM>
70Send back this contents instead of the <data> one. The num is set by:
71A) The test number in the request line is >10000 and this is the remainder
72of [test case number]%10000.
73B) The request was HTTP and included digest details, which adds 1000 to NUM
74C) If a HTTP request is NTLM type-1, it adds 1001 to num
75D) If a HTTP request is NTLM type-3, it adds 1002 to num
76E) If a HTTP request is Basic and num is already >=1000, it adds 1 to num
77
78Dynamically changing num in this way allows the test harness to be used to
79test authentication negotiation where several different requests must be sent
80to complete a transfer. The response to each request is found in its own data
81section. Validating the entire negotiation sequence can be done by
82specifying a datacheck section.
83</dataNUM>
84<connect>
85The connect section is used instead of the 'data' for all CONNECT
86requests. The remainder of the rules for the data section then apply but with
87a connect prefix.
88</connect>
89<datacheck [mode="text"] [nonewline="yes"]>
90if the data is sent but this is what should be checked afterwards. If
91'nonewline' is set, we will cut off the trailing newline of this given data
92before comparing with the one actually received by the client.
93
94Use the mode="text" attribute if the output is in text mode on platforms that
95have a text/binary difference.
96</datacheck>
97<datacheckNUM [nonewline="yes"] [mode="text"]>
98The contents of numbered datacheck sections are appended to the non-numbered
99one.
100</datacheckNUM>
101<size>
102number to return on a ftp SIZE command (set to -1 to make this command fail)
103</size>
104<mdtm>
105what to send back if the client sends a (FTP) MDTM command, set to -1 to
106have it return that the file doesn't exist
107</mdtm>
108<postcmd>
109special purpose server-command to control its behavior *after* the
110reply is sent
111For HTTP/HTTPS, these are supported:
112
113wait [secs]
114 - Pause for the given time
115</postcmd>
116<servercmd>
117Special-commands for the server.
118For FTP/SMTP/POP/IMAP, these are supported:
119
120REPLY [command] [return value] [response string]
121 - Changes how the server responds to the [command]. [response string] is
122 evaluated as a perl string, so it can contain embedded \r\n, for example.
123 There's a special [command] named "welcome" (without quotes) which is the
124 string sent immediately on connect as a welcome.
125COUNT [command] [num]
126 - Do the REPLY change for [command] only [num] times and then go back to the
127 built-in approach
128DELAY [command] [secs]
129 - Delay responding to this command for the given time
130RETRWEIRDO
131 - Enable the "weirdo" RETR case when multiple response lines appear at once
132 when a file is transferred
133RETRNOSIZE
134 - Make sure the RETR response doesn't contain the size of the file
135NOSAVE
136 - Don't actually save what is received
137SLOWDOWN
138 - Send FTP responses with 0.01 sec delay between each byte
139PASVBADIP
140 - makes PASV send back an illegal IP in its 227 response
141CAPA [capabilities]
142 - Enables support for and specifies a list of space separated capabilities to
143 return to the client for the IMAP CAPABILITY, POP3 CAPA and SMTP EHLO
144 commands
145AUTH [mechanisms]
146 - Enables support for SASL authentication and specifies a list of space
147 separated mechanisms for IMAP, POP3 and SMTP
148
149For HTTP/HTTPS:
150auth_required if this is set and a POST/PUT is made without auth, the
151 server will NOT wait for the full request body to get sent
152idle do nothing after receiving the request, just "sit idle"
153stream continuously send data to the client, never-ending
154writedelay: [secs] delay this amount between reply packets
155pipe: [num] tell the server to expect this many HTTP requests before
156 sending back anything, to allow pipelining tests
157skip: [num] instructs the server to ignore reading this many bytes from a PUT
158 or POST request
159
160rtp: part [num] channel [num] size [num]
161 stream a fake RTP packet for the given part on a chosen channel
162 with the given payload size
163
164connection-monitor When used, this will log [DISCONNECT] to the server.input
165 log when the connection is disconnected.
166upgrade when an HTTP upgrade header is found, the server will upgrade
167 to http2
168
169For TFTP:
170writedelay: [secs] delay this amount between reply packets (each packet being
171 512 bytes payload)
172</servercmd>
173</reply>
174
175<client>
176
177<server>
178What server(s) this test case requires/uses:
179
180file
181ftp
182ftp-ipv6
183ftps
184http
185http-ipv6
186http-pipe
187http-proxy
188http-unix
189https
190httptls+srp
191httptls+srp-ipv6
192http/2
193imap
194none
195pop3
196rtsp
197rtsp-ipv6
198scp
199sftp
200smtp
201socks4
202socks5
203
204Give only one per line. This subsection is mandatory.
205</server>
206
207<features>
208A list of features that MUST be present in the client/library for this test to
209be able to run. If a required feature is not present then the test will be
210SKIPPED.
211
212Alternatively a feature can be prefixed with an exclamation mark to indicate a
213feature is NOT required. If the feature is present then the test will be
214SKIPPED.
215
216Features testable here are:
217
218axTLS
219crypto
220debug
221getrlimit
222GnuTLS
223GSS-API
224http2
225idn
226ipv6
227Kerberos
228large_file
229libz
230Metalink
231NSS
232NTLM
233OpenSSL
234PSL
235socks
236SPNEGO
237SSL
238SSLpinning
239SSPI
240TLS-SRP
241TrackMemory
242unittest
243unix-sockets
244WinSSL
245
246as well as each protocol that curl supports. A protocol only needs to be
247specified if it is different from the server (useful when the server
248is 'none').
249</features>
250
251<killserver>
252Using the same syntax as in <server> but when mentioned here these servers
253are explicitly KILLED when this test case is completed. Only use this if there
254is no other alternatives. Using this of course requires subsequent tests to
255restart servers.
256</killserver>
257
258<precheck>
259A command line that if set gets run by the test script before the test. If an
260output is displayed by the command or if the return code is non-zero, the test
261will be skipped and the (single-line) output will be displayed as reason for
262not running the test. Variables are substituted as in the <command> section.
263</precheck>
264
265<postcheck>
266A command line that if set gets run by the test script after the test. If
267the command exists with a non-zero status code, the test will be considered
268to have failed. Variables are substituted as in the <command> section.
269</postcheck>
270
271<tool>
272Name of tool to use instead of "curl". This tool must be built and exist
273either in the libtest/ directory (if the tool starts with 'lib') or in the
274unit/ directory (if the tool starts with 'unit').
275</tool>
276
277<name>
278test case description
279</name>
280
281<setenv>
282variable1=contents1
283variable2=contents2
284
285Set the given environment variables to the specified value before the actual
286command is run. They are cleared again after the command has been run.
287Variables are first substituted as in the <command> section.
288</setenv>
289
290<command [option="no-output/no-include"] [timeout="secs"] [delay="secs"]
291 [type="perl"]>
292command line to run, there's a bunch of %variables that get replaced
293accordingly.
294
295Note that the URL that gets passed to the server actually controls what data
296that is returned. The last slash in the URL must be followed by a number. That
297number (N) will be used by the test-server to load test case N and return the
298data that is defined within the <reply><data></data></reply> section.
299
300If there's no test number found above, the HTTP test server will use the
301number following the last dot in the given hostname (made so that a CONNECT
302can still pass on test number) so that "foo.bar.123" gets treated as test case
303123. Alternatively, if an IPv6 address is provided to CONNECT, the last
304hexadecimal group in the address will be used as the test number! For example
305the address "[1234::ff]" would be treated as test case 255.
306
307Set type="perl" to write the test case as a perl script. It implies that
308there's no memory debugging and valgrind gets shut off for this test.
309
310Set option="no-output" to prevent the test script to slap on the --output
311argument that directs the output to a file. The --output is also not added if
312the verify/stdout section is used.
313
314Set option="no-include" to prevent the test script to slap on the --include
315argument.
316
317Set timeout="secs" to override default server logs advisor read lock timeout.
318This timeout is used by the test harness, once that the command has completed
319execution, to wait for the test server to write out server side log files and
320remove the lock that advised not to read them. The "secs" parameter is the not
321negative integer number of seconds for the timeout. This 'timeout' attribute
322is documented for completeness sake, but is deep test harness stuff and only
323needed for very singular and specific test cases. Avoid using it.
324
325Set delay="secs" to introduce a time delay once that the command has completed
326execution and before the <postcheck> section runs. The "secs" parameter is the
327not negative integer number of seconds for the delay. This 'delay' attribute
328is intended for very specific test cases, and normally not needed.
329
330Available substitute variables include:
331%CLIENT6IP - IPv6 address of the client running curl
332%CLIENTIP - IPv4 address of the client running curl
333%CURL - Path to the curl executable
334%FTP2PORT - Port number of the FTP server 2
335%FTP6PORT - IPv6 port number of the FTP server
336%FTPPORT - Port number of the FTP server
337%FTPSPORT - Port number of the FTPS server
338%FTPTIME2 - Timeout in seconds that should be just sufficient to receive
339 a response from the test FTP server
340%FTPTIME3 - Even longer than %FTPTIME2
341%GOPHER6PORT - IPv6 port number of the Gopher server
342%GOPHERPORT - Port number of the Gopher server
343%HOST6IP - IPv6 address of the host running this test
344%HOSTIP - IPv4 address of the host running this test
345%HTTP6PORT - IPv6 port number of the HTTP server
346%HTTPPIPEPORT - Port number of the HTTP pipelining server
347%HTTPUNIXPATH - Path to the Unix socket of the HTTP server
348%HTTPPORT - Port number of the HTTP server
349%HTTPSPORT - Port number of the HTTPS server
350%HTTPTLS6PORT - IPv6 port number of the HTTP TLS server
351%HTTPTLSPORT - Port number of the HTTP TLS server
352%IMAP6PORT - IPv6 port number of the IMAP server
353%IMAPPORT - Port number of the IMAP server
354%POP36PORT - IPv6 port number of the POP3 server
355%POP3PORT - Port number of the POP3 server
356%PROXYPORT - Port number of the HTTP proxy
357%PWD - Current directory
358%RTSP6PORT - IPv6 port number of the RTSP server
359%RTSPPORT - Port number of the RTSP server
360%SMTP6PORT - IPv6 port number of the SMTP server
361%SMTPPORT - Port number of the SMTP server
362%SOCKSPORT - Port number of the SOCKS4/5 server
363%SRCDIR - Full path to the source dir
364%SSHPORT - Port number of the SCP/SFTP server
365%TFTP6PORT - IPv6 port number of the TFTP server
366%TFTPPORT - Port number of the TFTP server
367%USER - Login ID of the user running the test
368</command>
369
370<file name="log/filename">
371This creates the named file with this content before the test case is run,
372which is useful if the test case needs a file to act on.
373Variables are substituted on the contents of the file as in the <command>
374section.
375</file>
376
377<stdin [nonewline="yes"]>
378Pass this given data on stdin to the tool.
379
380If 'nonewline' is set, we will cut off the trailing newline of this given data
381before comparing with the one actually received by the client
382</stdin>
383
384</client>
385
386<verify>
387<errorcode>
388numerical error code curl is supposed to return. Specify a list of accepted
389error codes by separating multiple numbers with comma. See test 237 for an
390example.
391</errorcode>
392<strip>
393One regex per line that is removed from the protocol dumps before the
394comparison is made. This is very useful to remove dependencies on dynamically
395changing protocol data such as port numbers or user-agent strings.
396</strip>
397<strippart>
398One perl op per line that operates on the protocol dump. This is pretty
399advanced. Example: "s/^EPRT .*/EPRT stripped/"
400</strippart>
401
402<protocol [nonewline="yes"]>
403
404the protocol dump curl should transmit, if 'nonewline' is set, we will cut off
405the trailing newline of this given data before comparing with the one actually
406sent by the client Variables are substituted as in the <command> section. The
407<strip> and <strippart> rules are applied before comparisons are made.
408
409</protocol>
410
411<proxy [nonewline="yes"]>
412
413The protocol dump curl should transmit to a HTTP proxy (when the http-proxy
414server is used), if 'nonewline' is set, we will cut off the trailing newline
415of this given data before comparing with the one actually sent by the client
416Variables are substituted as in the <command> section. The <strip> and
417<strippart> rules are applied before comparisons are made.
418
419</proxy>
420
421<stdout [mode="text"] [nonewline="yes"]>
422This verifies that this data was passed to stdout. Variables are
423substituted as in the <command> section.
424
425Use the mode="text" attribute if the output is in text mode on platforms that
426have a text/binary difference.
427
428If 'nonewline' is set, we will cut off the trailing newline of this given data
429before comparing with the one actually received by the client
430</stdout>
431<file name="log/filename" [mode="text"]>
432The file's contents must be identical to this after the test is complete.
433Use the mode="text" attribute if the output is in text mode on platforms that
434have a text/binary difference.
435Variables are substituted as in the <command> section.
436</file>
437<stripfile>
438One perl op per line that operates on the output file or stdout before being
439compared with what is stored in the test file. This is pretty
440advanced. Example: "s/^EPRT .*/EPRT stripped/"
441</stripfile>
442<upload>
443the contents of the upload data curl should have sent
444</upload>
445<valgrind>
446disable - disables the valgrind log check for this test
447</valgrind>
448</verify>
449
450</testcase>