Hey there!
I'm stuck with cookies usage. Today I sacrificed whole day but did not reached the result...
I'm grabbing TV programmes from tv.yandex.ru. The thing is that depending on where you're from, they either show some programmes or not. However if you manually set your location to Moscow in my case, this is saved to cookies. I've exported all cookies from this side and by experiment found out that following two lines are responsible for location:
.yandex.ru TRUE / FALSE 1889610884 yp 1605786884.ygu.0#1605786884.ygo.11475:213
.yandex.ru TRUE / FALSE 1668858884 yandex_gid 213
All cookies are in attached zip. And for the sake of simplicity I have no problem to use all lines. To test if they are working, I've tried curl without cookie:
➜ curl https://tv.yandex.ru/channels/334\?date\=2019-11-21\&period\=all-day
Moved Permanently. Redirecting to /?date=2019-11-21&period=all-day%
No success... Then I've tried curl with cookie:
➜ Downloads curl --cookie cookies.txt https://tv.yandex.ru/channels/334\?date\=2019-11-21\&period\=all-day
Moved Permanently. Redirecting to /channel/moskva-24-73?date=2019-11-21&period=all-day%
Redirect works just fine. It means cookie is correct. Now I've tried to use cookie with tv.yandex.ru.ini by enabling line:
site {loadcookie=cookies.txt}
I'm trying to run wg++:
[/share/Public/wg++] # ./run.sh
WebGrab+Plus/w MDB & REX Postprocess -- version V2.1.11.0
Jan van Straaten
Francis De Paemeleere
thanks to Paul Weterings and all the contributing users
--------------------------------------------------------------------------------
Job started at 20/11/2019 21:45:04
found: /share/Public/wg++/./siteini.user/tv.yandex.ru.ini -- Revision 07
processing /share/Public/wg++/temp_guide.xml ...
Found existing channel (xmltv_id=Москва 24) in the config file
...
i=index .=same c=change g=gab r=replace n=new
Group (0) :
update requested for - 1 - out of - 1 - channels for 9 day(s)
( 1/1 ) TV.YANDEX.RU -- chan. (xmltv_id=Москва 24) -- mode Incremental
cannot find /share/Public/wg++/./cookies.txt !
loadcookie failed! ... cannot update this channel
Job finished at 20/11/2019 21:45:05 done in 1s
[/share/Public/wg++] # ls
bin/ mdb/ run.sh* temp_guide.xml WebGrab++.config.backup20181017.xml WebGrab++.config copy.xml
cookies.txt README.md siteini.pack/ WebGrab++.config.backup20171107.xml WebGrab++.config.backup20190415.xml WebGrab++.config.example.xml
doc/ rex/ SiteIni.Pack.Update.sh* WebGrab++.config.backup20180310.xml WebGrab++.config.backup20191113.xml WebGrab++.config.xml
install.sh* robots/ siteini.user/ WebGrab++.config.backup20181010.xml WebGrab++.config.backup20191117.xml WebGrab++.log.txt
[/share/Public/wg++] #
As you can see log say's can't load cookie, as it's not there. However from ls shows that file is there for sure. Full log together with config file is attached in zip file.
Versions:
- wg++ V2.1.11.0
- mono 5.11.0
- Linux 4.14.24-qnap x86_64 GNU/Linux
I've tried older versions of wg++, however there's no difference. Any thoughts on what am I doing wrong?
Thanks in advance.
Fly
If cookie fail to load means you do not have the right path set, so try to set full path in siteini
Nope, I can't set full path, as then I'll get following error:
cannot find /share/CACHEDEV1_DATA/Public/wg++/.//share/CACHEDEV1_DATA/Public/wg++/share/Public/wg++/cookies.txt !
loadcookie failed! ... cannot update this channel
Seems that wg++ doesn't like full paths.
In case I create an empty cookies.txt file, It doesn't say there's a problem to load it. I suspect that a problem could be:
- either domain, as lines responsible for region are for domain ".yandex.ru", as EPG is grabbed from "tv.yandex.ru"
- or structure of cookie file (very unlikely, as I've compared with a structure from file "hot_cookies.txt" which is created by wg++ and structure matches
Any other ideas how to move forward, especially with domains assumption?
As a workaround I've started using proxy server in Moscow. In such case I do not need cookies.
I've figured out how to use cookie in tv.yandex.ru.ini
instead of using 'loadcookie=cookie.txt' which gives us error mentioned you should use it in following way:
site {ratingsystem=RU|episodesystem=onscreen|loadcookie=cookie.txt}