cURL: Auf einer ASP-Seite Formular befüllen und Ergebnis auslesen

[danimoe]

Hallo Forenmitglieder,

Ich habe ein Problem: Und zwar will ich auf TUIfly.com das Formular ausfüllen für die Suche von Flügen und das Ergebnis dann verwerten.

Soweit so gut. Ich versuche in der POST-Anweisung den VIEWSTATE mit zu übergeben. Ich bekomm leider nur die Standardseite mit unausgefüllten Feldern und ohne Stylesheets und JS-Dateien (ist ja klar, glaub ich).

PHP-Code:

curl_login('http://www.tuifly.com/Select.aspx','__EVENTTARGET=ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24LinkButtonNewSearch&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUBMGRk8c6YBCrCSJBXIOvLBnRhqSOpiIE%3D&scorigin=CS-Home&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24RadioButtonMarketStructure=RoundTrip&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketOrigin1=BRI&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketDestination1=CGN&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketDay1=09&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketMonth1=2009-07&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketDay2=28&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketMonth2=2009-07&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListNewPassengerType_ADT=1&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListPassengerType_INFANT=0&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListPassengerType_CHDD=0&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListPassengerType_CHD=0&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListPassengerType_ADT=1&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketDateRange1=2%7C2&ControlGroupAvailabilitySearchSelectView%24AvailabilitySearchInputSelectView%24DropDownListMarketDateRange2=2%7C2&ControlGroupSelectView%24AvailabilityInputSelectView','','off');

function curl_login($url,$data,$proxy,$proxystatus){
    $cookiefile = $_SERVER['REMOTE_ADDR'] . '.txt';
    $fp = fopen($cookiefile, "w");
    fclose($fp);
    $login = curl_init();
    curl_setopt($login, CURLOPT_COOKIEJAR, $cookiefile);
    curl_setopt($login, CURLOPT_COOKIEFILE, $cookiefile);
    curl_setopt($login, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);//"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
    curl_setopt($login, CURLOPT_TIMEOUT, 40);
    curl_setopt($login, CURLOPT_RETURNTRANSFER, TRUE);
    if ($proxystatus == 'on') {
        curl_setopt($login, CURLOPT_SSL_VERIFYHOST, FALSE);
        curl_setopt($login, CURLOPT_HTTPPROXYTUNNEL, TRUE);
        curl_setopt($login, CURLOPT_PROXY, $proxy);
    }
    curl_setopt($login, CURLOPT_URL, $url);
    curl_setopt($login, CURLOPT_HEADER, FALSE);
    curl_setopt($login, CURLOPT_HTTPHEADER, Array('Expect: '));
    curl_setopt($login, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
    curl_setopt($login, CURLOPT_FOLLOWLOCATION, TRUE);
    curl_setopt($login, CURLOPT_POST, TRUE);
    curl_setopt($login, CURLOPT_POSTFIELDS, $data);
    ob_start();      // prevent any output
    $ret = curl_exec ($login); // execute the curl command
    ob_end_clean();  // stop preventing output
    curl_close ($login);
    unset($login);
    
    $fp = fopen($cookiefile, 'r');
    while ($line = fgets($fp)) {
        $line = preg_replace('/\s*#.*$/', '', $line);
        if (trim($line) == '') continue;
        //echo $line;
        $fields = explode("\t", trim($line));
        $host = explode('.', $fields[0]);
        //array_shift($host);
        $host = implode('.', $host);
        //var_dump($fields);
        //setcookie($fields[5], $fields[6], $fields[4], $fields[2], $host, $fields[3] == 'TRUE');
        header('Set-Cookie: '.rawurlencode($fields[5]).'='.rawurlencode($fields[6]).'; Domain=www.tuifly.de; Path=/', false);
    }
    fclose($fp);
    
    return $ret;
}
 
function curl_grab_page($site,$proxy,$proxystatus){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
    if ($proxystatus == 'on') {
        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
        curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
        curl_setopt($ch, CURLOPT_PROXY, $proxy);
    }
    curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
    curl_setopt($ch, CURLOPT_URL, $site);
    ob_start();      // prevent any output
    $ret = curl_exec ($ch); // execute the curl command
    ob_end_clean();  // stop preventing output
    curl_close ($ch);
    return $ret;
} 


[danimoe]

Dies ist der HTTP-Request, wenn ich auf Suchen klicke:

http://www.tuifly.com/CompactSearchSSI.aspx?culture=de-DE

POST /CompactSearchSSI.aspx?culture=de-DE HTTP/1.1
Host: www.tuifly.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://www.tuifly.com/de/
Cookie: CP=null*; s_nr=1246880079534; s_lastvisit=1246892990555; s_vi=[CS]v1|4A51DC6F00004603-A3A084E00002679[CE]; AvailabilitySearchData=UnxCUkl8Q0dOfDEwfDIwMDktMDd8Mjh8MjAwOS0wN3wx; POPUPCHECK=1246966358458; s_sq=tuifly-produktion-de%3D%2526pid%253Dde/index.html%2526pidt%253D1%2526oid%253Dfunction%252528%252529%25257B%25257D%2526oidt%253D2%2526ot%253DA; s_cc=true; sID=skysales.sky05; fID=farefinder.sky01; ASP.NET_SessionId=nwmitiumg0yuk2ibqvt0ayzv
Content-Type: application/x-www-form-urlencoded
Content-Length: 1989
__EVENTTARGET=ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24LinkButtonNewSearch&__EVENTARGUMENT=&scorigin=CS-Home&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24RadioButtonMarketStructure=RoundTrip&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketOrigin1=BRI&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketDestination1=CGN&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketDay1=10&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketMonth1=2009-07&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketDay2=28&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketMonth2=2009-07&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListNewPassengerType_ADT=1&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListPassengerType_INFANT=0&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListPassengerType_CHDD=0&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListPassengerType_CHD=0&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListPassengerType_ADT=1&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketDateRange1=2%7C2&ControlGroupCompactSearchSSIView%24AvailabilitySearchInputCompactSearchSSIView%24DropDownListMarketDateRange2=2%7C2&checkinLastname=&checkinBid=&language=de&ControlGroupCompactSearchSSIView%24BookingRetrieveInputComapctSearchSSIView%24CONFIRMATIONNUMBER1=&ControlGroupCompactSearchSSIView%24BookingRetrieveInputComapctSearchSSIView%24CONTACTEMAIL1=&cf=CompactSearch
HTTP/1.x 302 Found
Date: Mon, 06 Jul 2009 15:18:24 GMT
Server: Microsoft-IIS/6.0
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml", CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Pragma: no-cache, no-cache, no-cache
Location: /Select.aspx
Cache-Control: no-cache, no-store
Expires: -1
Content-Type: text/html; charset=utf-8
Set-Cookie: sID=skysales.sky05;path=/
Set-Cookie: AvailabilitySearchData=UnxCUkl8Q0dOfDEwfDIwMDktMDd8Mjh8MjAwOS0wN3wx; expires=Wed, 05-Aug-2009 15:18:24 GMT; path=/
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-UA-Compatible: IE=EmulateIE7
Content-Length: 121
Proxy-Connection: Keep-alive
----------------------------------------------------------
http://www.tuifly.com/Select.aspx

GET /Select.aspx HTTP/1.1
Host: www.tuifly.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://www.tuifly.com/de/
Cookie: CP=null*; s_nr=1246880079534; s_lastvisit=1246892990555; s_vi=[CS]v1|4A51DC6F00004603-A3A084E00002679[CE]; AvailabilitySearchData=UnxCUkl8Q0dOfDEwfDIwMDktMDd8Mjh8MjAwOS0wN3wx; POPUPCHECK=1246966358458; s_sq=tuifly-produktion-de%3D%2526pid%253Dde/index.html%2526pidt%253D1%2526oid%253Dfunction%252528%252529%25257B%25257D%2526oidt%253D2%2526ot%253DA; s_cc=true; sID=skysales.sky05; fID=farefinder.sky01; ASP.NET_SessionId=nwmitiumg0yuk2ibqvt0ayzv

HTTP/1.x 200 OK
Date: Mon, 06 Jul 2009 15:18:25 GMT
Server: Microsoft-IIS/6.0
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml", CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Pragma: no-cache, no-cache, no-cache
Cache-Control: no-cache, no-store
Expires: -1
Content-Type: text/html; charset=utf-8
Set-Cookie: sID=skysales.sky05;path=/
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-UA-Compatible: IE=EmulateIE7
Content-Length: 36944
Proxy-Connection: Keep-alive
----------------------------------------------------------
http://www.tuifly.com/TaxAndFeeInclusiveDisplay-resource.aspx?flightKeys=T~TTFLY~3500~~None|X3~3105~%20~~BRI~07/09/2009~CGN~07/10/2009,V~VTFLY~3500~~None|X3~3104~%20~~CGN~07/27/2009~BRI~07/27/2009

GET /TaxAndFeeInclusiveDisplay-resource.aspx?flightKeys=T~TTFLY~3500~~None|X3~3105~%20~~BRI~07/09/2009~CGN~07/10/2009,V~VTFLY~3500~~None|X3~3104~%20~~CGN~07/27/2009~BRI~07/27/2009 HTTP/1.1
Host: www.tuifly.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11
Accept: text/html, */*
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
X-Requested-With: XMLHttpRequest
Referer: http://www.tuifly.com/Select.aspx
Cookie: CP=null*; s_nr=1246880079534; s_lastvisit=1246892990555; s_vi=[CS]v1|4A51DC6F00004603-A3A084E00002679[CE]; AvailabilitySearchData=UnxCUkl8Q0dOfDEwfDIwMDktMDd8Mjh8MjAwOS0wN3wx; POPUPCHECK=1246966358458; s_sq=tuifly-produktion-de%3D%2526pid%253Dde/index.html%2526pidt%253D1%2526oid%253Dfunction%252528%252529%25257B%25257D%2526oidt%253D2%2526ot%253DA; s_cc=true; sID=skysales.sky05; fID=farefinder.sky01; ASP.NET_SessionId=nwmitiumg0yuk2ibqvt0ayzv

HTTP/1.x 200 OK
Date: Mon, 06 Jul 2009 15:18:27 GMT
Server: Microsoft-IIS/6.0
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml", CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Pragma: no-cache, no-cache, no-cache
Cache-Control: no-cache, no-store
Expires: -1
Content-Type: text/html; charset=utf-8
Set-Cookie: sID=skysales.sky05;path=/
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-UA-Compatible: IE=EmulateIE7
Content-Length: 934
Proxy-Connection: Keep-alive

Und dies beim normalen Aufruf:

http://www.tuifly.com/de/index.html

GET /de/index.html HTTP/1.1
Host: www.tuifly.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Cookie: CP=null*; s_nr=1246880079534; s_lastvisit=1246892990555; s_vi=[CS]v1|4A51DC6F00004603-A3A084E00002679[CE]; AvailabilitySearchData=UnxCUkl8Q0dOfDEwfDIwMDktMDd8Mjh8MjAwOS0wN3wx; POPUPCHECK=1246966358458; s_sq=%5B%5BB%5D%5D; s_cc=true; sID=skysales.sky05; fID=farefinder.sky01; ASP.NET_SessionId=nwmitiumg0yuk2ibqvt0ayzv

HTTP/1.x 200 OK
Date: Mon, 06 Jul 2009 15:20:35 GMT
Server: Apache
Last-Modified: Mon, 06 Jul 2009 15:10:30 GMT
Etag: "10ff5a5-3377c-46e0ae829d659"-gzip
Accept-Ranges: bytes
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-UA-Compatible: IE=EmulateIE7
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM", policyref="/w3c/p3p.xml"
Content-Length: 32550
Content-Type: text/html; charset=utf-8
Proxy-Connection: Keep-alive

Ich komm einfach nicht mehr weiter und hab schon einiges versucht, aber es scheint, dass ich nicht auf den Seitenaufbau dahintersteige und wie diese Suchanfrage genau funktioniert.

Wenn ich z.B. in den Browser die Adresse mit den Post eingebe, dann zeigt er mir auch die Seite mit den Ergebnissen an, bzw. erst wenn ich vorher schon mal gesucht habe. Aber es klappt auf jeden Fall.

Ich bin dankbar für jede Hilfe und auch nur kleine Hinweise sind mir absolut willkommen.