lh | 9ed821d | 2023-04-07 01:36:19 -0700 | [diff] [blame] | 1 | The test suite's file format is very simple and extensible, closely |
| 2 | resembling XML. All data for a single test case resides in a single |
| 3 | ASCII file. Labels mark the beginning and the end of all sections, and each |
| 4 | label must be written in its own line. Comments are either XML-style |
| 5 | (enclosed with <!-- and -->) or C-style (beginning with #) and must appear |
| 6 | on their own lines and not alongside actual test data. Most test data files |
| 7 | are syntactically valid XML, although a few files are not (lack of |
| 8 | support for character entities and the preservation of CR/LF characters at |
| 9 | the end of lines are the biggest differences). |
| 10 | |
| 11 | The file begins with a 'testcase' tag, which encompasses the remainder of |
| 12 | the file. |
| 13 | |
| 14 | <testcase> |
| 15 | |
| 16 | Each file is split up in three main sections: reply, client and verify. The |
| 17 | reply section is used for the server to know what to send as a reply for the |
| 18 | requests curl sends, the client section defines how the client should behave |
| 19 | while the verify section defines how to verify that the data stored after a |
| 20 | command has been run ended up correctly. |
| 21 | |
| 22 | Each main section has a number of available subsections that can be |
| 23 | specified, that will be checked/used if specified. This document includes all |
| 24 | the subsections currently supported. |
| 25 | |
| 26 | Main sections are 'info', 'reply', 'client' and 'verify'. |
| 27 | |
| 28 | <info> |
| 29 | <keywords> |
| 30 | A newline-separated list of keywords describing what this test case uses and |
| 31 | tests. Try to use an already used keyword. These keywords will be used for |
| 32 | statistical/informational purposes and for choosing or skipping classes |
| 33 | of tests. "Keywords" must begin with an alphabetic character, "-", "[" |
| 34 | or "{" and may actually consist of multiple words separated by spaces |
| 35 | which are treated together as a single identifier. |
| 36 | </keywords> |
| 37 | </info> |
| 38 | |
| 39 | <reply> |
| 40 | <data [nocheck="yes"] [sendzero="yes"] [base64="yes"]> |
| 41 | data to be sent to the client on its request and later verified that it arrived |
| 42 | safely. Set nocheck="yes" to prevent the test script from verifying the arrival |
| 43 | of this data. |
| 44 | |
| 45 | If the data contains 'swsclose' anywhere within the start and end tag, and |
| 46 | this is a HTTP test, then the connection will be closed by the server after |
| 47 | this response is sent. If not, the connection will be kept persistent. |
| 48 | |
| 49 | If the data contains 'swsbounce' anywhere within the start and end tag, the |
| 50 | HTTP server will detect if this is a second request using the same test and |
| 51 | part number and will then increase the part number with one. This is useful |
| 52 | for auth tests and similar. |
| 53 | |
| 54 | 'sendzero' set to yes means that the (FTP) server will "send" the data even if |
| 55 | the size is zero bytes. Used to verify curl's behaviour on zero bytes |
| 56 | transfers. |
| 57 | |
| 58 | 'base64' set to yes means that the data provided in the test-file is a chunk |
| 59 | of data encoded with base64. It is the only way a test case can contain binary |
| 60 | data. (This attribute can in fact be used on any section, but it doesn't make |
| 61 | much sense for other sections than "data"). |
| 62 | |
| 63 | For FTP file listings, the <data> section will be used *only* if you make sure |
| 64 | that there has been a CWD done first to a directory named 'test-[num]' where |
| 65 | [num] is the test case number. Otherwise the ftp server can't know from which |
| 66 | test file to load the list content. |
| 67 | |
| 68 | </data> |
| 69 | <dataNUM> |
| 70 | Send back this contents instead of the <data> one. The num is set by: |
| 71 | A) The test number in the request line is >10000 and this is the remainder |
| 72 | of [test case number]%10000. |
| 73 | B) The request was HTTP and included digest details, which adds 1000 to NUM |
| 74 | C) If a HTTP request is NTLM type-1, it adds 1001 to num |
| 75 | D) If a HTTP request is NTLM type-3, it adds 1002 to num |
| 76 | E) If a HTTP request is Basic and num is already >=1000, it adds 1 to num |
| 77 | |
| 78 | Dynamically changing num in this way allows the test harness to be used to |
| 79 | test authentication negotiation where several different requests must be sent |
| 80 | to complete a transfer. The response to each request is found in its own data |
| 81 | section. Validating the entire negotiation sequence can be done by |
| 82 | specifying a datacheck section. |
| 83 | </dataNUM> |
| 84 | <connect> |
| 85 | The connect section is used instead of the 'data' for all CONNECT |
| 86 | requests. The remainder of the rules for the data section then apply but with |
| 87 | a connect prefix. |
| 88 | </connect> |
| 89 | <datacheck [mode="text"] [nonewline="yes"]> |
| 90 | if the data is sent but this is what should be checked afterwards. If |
| 91 | 'nonewline' is set, we will cut off the trailing newline of this given data |
| 92 | before comparing with the one actually received by the client. |
| 93 | |
| 94 | Use the mode="text" attribute if the output is in text mode on platforms that |
| 95 | have a text/binary difference. |
| 96 | </datacheck> |
| 97 | <datacheckNUM [nonewline="yes"] [mode="text"]> |
| 98 | The contents of numbered datacheck sections are appended to the non-numbered |
| 99 | one. |
| 100 | </datacheckNUM> |
| 101 | <size> |
| 102 | number to return on a ftp SIZE command (set to -1 to make this command fail) |
| 103 | </size> |
| 104 | <mdtm> |
| 105 | what to send back if the client sends a (FTP) MDTM command, set to -1 to |
| 106 | have it return that the file doesn't exist |
| 107 | </mdtm> |
| 108 | <postcmd> |
| 109 | special purpose server-command to control its behavior *after* the |
| 110 | reply is sent |
| 111 | For HTTP/HTTPS, these are supported: |
| 112 | |
| 113 | wait [secs] |
| 114 | - Pause for the given time |
| 115 | </postcmd> |
| 116 | <servercmd> |
| 117 | Special-commands for the server. |
| 118 | For FTP/SMTP/POP/IMAP, these are supported: |
| 119 | |
| 120 | REPLY [command] [return value] [response string] |
| 121 | - Changes how the server responds to the [command]. [response string] is |
| 122 | evaluated as a perl string, so it can contain embedded \r\n, for example. |
| 123 | There's a special [command] named "welcome" (without quotes) which is the |
| 124 | string sent immediately on connect as a welcome. |
| 125 | COUNT [command] [num] |
| 126 | - Do the REPLY change for [command] only [num] times and then go back to the |
| 127 | built-in approach |
| 128 | DELAY [command] [secs] |
| 129 | - Delay responding to this command for the given time |
| 130 | RETRWEIRDO |
| 131 | - Enable the "weirdo" RETR case when multiple response lines appear at once |
| 132 | when a file is transferred |
| 133 | RETRNOSIZE |
| 134 | - Make sure the RETR response doesn't contain the size of the file |
| 135 | NOSAVE |
| 136 | - Don't actually save what is received |
| 137 | SLOWDOWN |
| 138 | - Send FTP responses with 0.01 sec delay between each byte |
| 139 | PASVBADIP |
| 140 | - makes PASV send back an illegal IP in its 227 response |
| 141 | CAPA [capabilities] |
| 142 | - Enables support for and specifies a list of space separated capabilities to |
| 143 | return to the client for the IMAP CAPABILITY, POP3 CAPA and SMTP EHLO |
| 144 | commands |
| 145 | AUTH [mechanisms] |
| 146 | - Enables support for SASL authentication and specifies a list of space |
| 147 | separated mechanisms for IMAP, POP3 and SMTP |
| 148 | |
| 149 | For HTTP/HTTPS: |
| 150 | auth_required if this is set and a POST/PUT is made without auth, the |
| 151 | server will NOT wait for the full request body to get sent |
| 152 | idle do nothing after receiving the request, just "sit idle" |
| 153 | stream continuously send data to the client, never-ending |
| 154 | writedelay: [secs] delay this amount between reply packets |
| 155 | pipe: [num] tell the server to expect this many HTTP requests before |
| 156 | sending back anything, to allow pipelining tests |
| 157 | skip: [num] instructs the server to ignore reading this many bytes from a PUT |
| 158 | or POST request |
| 159 | |
| 160 | rtp: part [num] channel [num] size [num] |
| 161 | stream a fake RTP packet for the given part on a chosen channel |
| 162 | with the given payload size |
| 163 | |
| 164 | connection-monitor When used, this will log [DISCONNECT] to the server.input |
| 165 | log when the connection is disconnected. |
| 166 | upgrade when an HTTP upgrade header is found, the server will upgrade |
| 167 | to http2 |
| 168 | |
| 169 | For TFTP: |
| 170 | writedelay: [secs] delay this amount between reply packets (each packet being |
| 171 | 512 bytes payload) |
| 172 | </servercmd> |
| 173 | </reply> |
| 174 | |
| 175 | <client> |
| 176 | |
| 177 | <server> |
| 178 | What server(s) this test case requires/uses: |
| 179 | |
| 180 | file |
| 181 | ftp |
| 182 | ftp-ipv6 |
| 183 | ftps |
| 184 | http |
| 185 | http-ipv6 |
| 186 | http-pipe |
| 187 | http-proxy |
| 188 | http-unix |
| 189 | https |
| 190 | httptls+srp |
| 191 | httptls+srp-ipv6 |
| 192 | http/2 |
| 193 | imap |
| 194 | none |
| 195 | pop3 |
| 196 | rtsp |
| 197 | rtsp-ipv6 |
| 198 | scp |
| 199 | sftp |
| 200 | smtp |
| 201 | socks4 |
| 202 | socks5 |
| 203 | |
| 204 | Give only one per line. This subsection is mandatory. |
| 205 | </server> |
| 206 | |
| 207 | <features> |
| 208 | A list of features that MUST be present in the client/library for this test to |
| 209 | be able to run. If a required feature is not present then the test will be |
| 210 | SKIPPED. |
| 211 | |
| 212 | Alternatively a feature can be prefixed with an exclamation mark to indicate a |
| 213 | feature is NOT required. If the feature is present then the test will be |
| 214 | SKIPPED. |
| 215 | |
| 216 | Features testable here are: |
| 217 | |
| 218 | axTLS |
| 219 | crypto |
| 220 | debug |
| 221 | getrlimit |
| 222 | GnuTLS |
| 223 | GSS-API |
| 224 | http2 |
| 225 | idn |
| 226 | ipv6 |
| 227 | Kerberos |
| 228 | large_file |
| 229 | libz |
| 230 | Metalink |
| 231 | NSS |
| 232 | NTLM |
| 233 | OpenSSL |
| 234 | PSL |
| 235 | socks |
| 236 | SPNEGO |
| 237 | SSL |
| 238 | SSLpinning |
| 239 | SSPI |
| 240 | TLS-SRP |
| 241 | TrackMemory |
| 242 | unittest |
| 243 | unix-sockets |
| 244 | WinSSL |
| 245 | |
| 246 | as well as each protocol that curl supports. A protocol only needs to be |
| 247 | specified if it is different from the server (useful when the server |
| 248 | is 'none'). |
| 249 | </features> |
| 250 | |
| 251 | <killserver> |
| 252 | Using the same syntax as in <server> but when mentioned here these servers |
| 253 | are explicitly KILLED when this test case is completed. Only use this if there |
| 254 | is no other alternatives. Using this of course requires subsequent tests to |
| 255 | restart servers. |
| 256 | </killserver> |
| 257 | |
| 258 | <precheck> |
| 259 | A command line that if set gets run by the test script before the test. If an |
| 260 | output is displayed by the command or if the return code is non-zero, the test |
| 261 | will be skipped and the (single-line) output will be displayed as reason for |
| 262 | not running the test. Variables are substituted as in the <command> section. |
| 263 | </precheck> |
| 264 | |
| 265 | <postcheck> |
| 266 | A command line that if set gets run by the test script after the test. If |
| 267 | the command exists with a non-zero status code, the test will be considered |
| 268 | to have failed. Variables are substituted as in the <command> section. |
| 269 | </postcheck> |
| 270 | |
| 271 | <tool> |
| 272 | Name of tool to use instead of "curl". This tool must be built and exist |
| 273 | either in the libtest/ directory (if the tool starts with 'lib') or in the |
| 274 | unit/ directory (if the tool starts with 'unit'). |
| 275 | </tool> |
| 276 | |
| 277 | <name> |
| 278 | test case description |
| 279 | </name> |
| 280 | |
| 281 | <setenv> |
| 282 | variable1=contents1 |
| 283 | variable2=contents2 |
| 284 | |
| 285 | Set the given environment variables to the specified value before the actual |
| 286 | command is run. They are cleared again after the command has been run. |
| 287 | Variables are first substituted as in the <command> section. |
| 288 | </setenv> |
| 289 | |
| 290 | <command [option="no-output/no-include"] [timeout="secs"] [delay="secs"] |
| 291 | [type="perl"]> |
| 292 | command line to run, there's a bunch of %variables that get replaced |
| 293 | accordingly. |
| 294 | |
| 295 | Note that the URL that gets passed to the server actually controls what data |
| 296 | that is returned. The last slash in the URL must be followed by a number. That |
| 297 | number (N) will be used by the test-server to load test case N and return the |
| 298 | data that is defined within the <reply><data></data></reply> section. |
| 299 | |
| 300 | If there's no test number found above, the HTTP test server will use the |
| 301 | number following the last dot in the given hostname (made so that a CONNECT |
| 302 | can still pass on test number) so that "foo.bar.123" gets treated as test case |
| 303 | 123. Alternatively, if an IPv6 address is provided to CONNECT, the last |
| 304 | hexadecimal group in the address will be used as the test number! For example |
| 305 | the address "[1234::ff]" would be treated as test case 255. |
| 306 | |
| 307 | Set type="perl" to write the test case as a perl script. It implies that |
| 308 | there's no memory debugging and valgrind gets shut off for this test. |
| 309 | |
| 310 | Set option="no-output" to prevent the test script to slap on the --output |
| 311 | argument that directs the output to a file. The --output is also not added if |
| 312 | the verify/stdout section is used. |
| 313 | |
| 314 | Set option="no-include" to prevent the test script to slap on the --include |
| 315 | argument. |
| 316 | |
| 317 | Set timeout="secs" to override default server logs advisor read lock timeout. |
| 318 | This timeout is used by the test harness, once that the command has completed |
| 319 | execution, to wait for the test server to write out server side log files and |
| 320 | remove the lock that advised not to read them. The "secs" parameter is the not |
| 321 | negative integer number of seconds for the timeout. This 'timeout' attribute |
| 322 | is documented for completeness sake, but is deep test harness stuff and only |
| 323 | needed for very singular and specific test cases. Avoid using it. |
| 324 | |
| 325 | Set delay="secs" to introduce a time delay once that the command has completed |
| 326 | execution and before the <postcheck> section runs. The "secs" parameter is the |
| 327 | not negative integer number of seconds for the delay. This 'delay' attribute |
| 328 | is intended for very specific test cases, and normally not needed. |
| 329 | |
| 330 | Available substitute variables include: |
| 331 | %CLIENT6IP - IPv6 address of the client running curl |
| 332 | %CLIENTIP - IPv4 address of the client running curl |
| 333 | %CURL - Path to the curl executable |
| 334 | %FTP2PORT - Port number of the FTP server 2 |
| 335 | %FTP6PORT - IPv6 port number of the FTP server |
| 336 | %FTPPORT - Port number of the FTP server |
| 337 | %FTPSPORT - Port number of the FTPS server |
| 338 | %FTPTIME2 - Timeout in seconds that should be just sufficient to receive |
| 339 | a response from the test FTP server |
| 340 | %FTPTIME3 - Even longer than %FTPTIME2 |
| 341 | %GOPHER6PORT - IPv6 port number of the Gopher server |
| 342 | %GOPHERPORT - Port number of the Gopher server |
| 343 | %HOST6IP - IPv6 address of the host running this test |
| 344 | %HOSTIP - IPv4 address of the host running this test |
| 345 | %HTTP6PORT - IPv6 port number of the HTTP server |
| 346 | %HTTPPIPEPORT - Port number of the HTTP pipelining server |
| 347 | %HTTPUNIXPATH - Path to the Unix socket of the HTTP server |
| 348 | %HTTPPORT - Port number of the HTTP server |
| 349 | %HTTPSPORT - Port number of the HTTPS server |
| 350 | %HTTPTLS6PORT - IPv6 port number of the HTTP TLS server |
| 351 | %HTTPTLSPORT - Port number of the HTTP TLS server |
| 352 | %IMAP6PORT - IPv6 port number of the IMAP server |
| 353 | %IMAPPORT - Port number of the IMAP server |
| 354 | %POP36PORT - IPv6 port number of the POP3 server |
| 355 | %POP3PORT - Port number of the POP3 server |
| 356 | %PROXYPORT - Port number of the HTTP proxy |
| 357 | %PWD - Current directory |
| 358 | %RTSP6PORT - IPv6 port number of the RTSP server |
| 359 | %RTSPPORT - Port number of the RTSP server |
| 360 | %SMTP6PORT - IPv6 port number of the SMTP server |
| 361 | %SMTPPORT - Port number of the SMTP server |
| 362 | %SOCKSPORT - Port number of the SOCKS4/5 server |
| 363 | %SRCDIR - Full path to the source dir |
| 364 | %SSHPORT - Port number of the SCP/SFTP server |
| 365 | %TFTP6PORT - IPv6 port number of the TFTP server |
| 366 | %TFTPPORT - Port number of the TFTP server |
| 367 | %USER - Login ID of the user running the test |
| 368 | </command> |
| 369 | |
| 370 | <file name="log/filename"> |
| 371 | This creates the named file with this content before the test case is run, |
| 372 | which is useful if the test case needs a file to act on. |
| 373 | Variables are substituted on the contents of the file as in the <command> |
| 374 | section. |
| 375 | </file> |
| 376 | |
| 377 | <stdin [nonewline="yes"]> |
| 378 | Pass this given data on stdin to the tool. |
| 379 | |
| 380 | If 'nonewline' is set, we will cut off the trailing newline of this given data |
| 381 | before comparing with the one actually received by the client |
| 382 | </stdin> |
| 383 | |
| 384 | </client> |
| 385 | |
| 386 | <verify> |
| 387 | <errorcode> |
| 388 | numerical error code curl is supposed to return. Specify a list of accepted |
| 389 | error codes by separating multiple numbers with comma. See test 237 for an |
| 390 | example. |
| 391 | </errorcode> |
| 392 | <strip> |
| 393 | One regex per line that is removed from the protocol dumps before the |
| 394 | comparison is made. This is very useful to remove dependencies on dynamically |
| 395 | changing protocol data such as port numbers or user-agent strings. |
| 396 | </strip> |
| 397 | <strippart> |
| 398 | One perl op per line that operates on the protocol dump. This is pretty |
| 399 | advanced. Example: "s/^EPRT .*/EPRT stripped/" |
| 400 | </strippart> |
| 401 | |
| 402 | <protocol [nonewline="yes"]> |
| 403 | |
| 404 | the protocol dump curl should transmit, if 'nonewline' is set, we will cut off |
| 405 | the trailing newline of this given data before comparing with the one actually |
| 406 | sent by the client Variables are substituted as in the <command> section. The |
| 407 | <strip> and <strippart> rules are applied before comparisons are made. |
| 408 | |
| 409 | </protocol> |
| 410 | |
| 411 | <proxy [nonewline="yes"]> |
| 412 | |
| 413 | The protocol dump curl should transmit to a HTTP proxy (when the http-proxy |
| 414 | server is used), if 'nonewline' is set, we will cut off the trailing newline |
| 415 | of this given data before comparing with the one actually sent by the client |
| 416 | Variables are substituted as in the <command> section. The <strip> and |
| 417 | <strippart> rules are applied before comparisons are made. |
| 418 | |
| 419 | </proxy> |
| 420 | |
| 421 | <stdout [mode="text"] [nonewline="yes"]> |
| 422 | This verifies that this data was passed to stdout. Variables are |
| 423 | substituted as in the <command> section. |
| 424 | |
| 425 | Use the mode="text" attribute if the output is in text mode on platforms that |
| 426 | have a text/binary difference. |
| 427 | |
| 428 | If 'nonewline' is set, we will cut off the trailing newline of this given data |
| 429 | before comparing with the one actually received by the client |
| 430 | </stdout> |
| 431 | <file name="log/filename" [mode="text"]> |
| 432 | The file's contents must be identical to this after the test is complete. |
| 433 | Use the mode="text" attribute if the output is in text mode on platforms that |
| 434 | have a text/binary difference. |
| 435 | Variables are substituted as in the <command> section. |
| 436 | </file> |
| 437 | <stripfile> |
| 438 | One perl op per line that operates on the output file or stdout before being |
| 439 | compared with what is stored in the test file. This is pretty |
| 440 | advanced. Example: "s/^EPRT .*/EPRT stripped/" |
| 441 | </stripfile> |
| 442 | <upload> |
| 443 | the contents of the upload data curl should have sent |
| 444 | </upload> |
| 445 | <valgrind> |
| 446 | disable - disables the valgrind log check for this test |
| 447 | </valgrind> |
| 448 | </verify> |
| 449 | |
| 450 | </testcase> |