grab from enigma2 openwebif

;) well, i want to import to TVHeadend, but therefore i 1st need the data as xml ;)

here is a sample from epg.dat from enigma2, its always a full epg ~ 12 mb ... sorry.

https://1drv.ms/u/s!ApY1s9BRY-iSmWzeu2oP7bUS845o

too bad, already thought so, i also opened it and its some code in there ...

my intention, i also have a TVHeadend Server besides the enigma2 server, TVH now uses the enigma2 box as iptv source,

so then i dont have any OTA EPG anymore in TVH. WG+ i only used for the channels i am recording Series cause i like to have the season, episode infos,

on the enigma2 Box its already all there, EPG with series infos and so on ... so why double handling when its there already ;)

And, i couldnt find a decent source to add all channels with WG+ to TVH ... always some missing and not really reliable here, i need pretty much all german

HDTV and UHD epg´s, also including sports ... Kabel Deustchland (Vodafone, this grabber works ok), HD+ and Sky Satellite.

well, i ll try some more sites and see if i can find a good source for my channels epg´s ;)

thanks for taking a look

i also checked at the /web/epg.... interface, figured i had to replace all ":; ;..."

so a sample looks like this here

http://192.168.1.103/web/epgservice?sRef=1%3A0%3A19%3A6E%3AD%3A85%3AFFFF...

i take a look at the sample.ini to see if that works

edit

ok, 1st tests where not successful

.ini

site {url=openwebif|timezone=UTC+01:00|maxdays=6.1|cultureinfo=de-GB|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="4">2</retry>}

urldate.format {daycounter|0}
url_index{url|http://root:password@192.168.1.103/web/epgservice?sRef=|channel|}

index_showsplit.scrub {multi (debug)|<table style=||</table>}
index_start.scrub {single (includeblock=2)|<td>||</td>|}
index_stop.scrub {single ()|<td valign="top">||</td>|}
index_title.scrub {single (includeblock=1)|<td>||</td>|}
index_subtitle.scrub {single (includeblock=4)|<td>||</td>|}
index_description.scrub {single ()|<td colspan="2" rowspan="2">||</td>|}

.xml

<settings>

  <filename>test.xml</filename>

<!--  

  <mode>m, n, f</mode>

 --> 

  <mode>m, n, #</mode> 

  <proxy>automatic</proxy> 

  <logging>on</logging>

  <retry>3</retry>

  <skip>13, 1</skip>

  <user-agent>random</user-agent>

  <timespan>5</timespan>

   <!-- update mode

  i or incremental     only updates of changes , gabs, repairs and new shows

  l or light           forces update of today and new shows, rest as incremental

  s or smart           forces udate of today and tomorrow and new shows, rest as light

  f or full or force   forces full update 

  If one of these values is entered here it will apply to all channels selected for update 

  (see channel).  This value overrules the value of 'update' for in the individual channels 

  If no value is entered here the individual 'update' values from the channellist are taken -->

  <update>f</update>

  <!--Polen-->

    <channel update="f" site="openwebif" site_id="1:0:19:30D7:413:1:C00000:0:0:0:" xmltv_id="Eurosport 2 HD Xtra">Eurosport 2 HD Xtra</channel>

<channel update="f" site="openwebif" site_id="1:0:19:7C:6:85:FFFF0000:0:0:0:" xmltv_id="Fox HD">Fox HD</channel>

</settings>

respond

--------------------------------------------------------------------------------

Job started at 11/02/2018 14:41:19
timezone=UTC+01:00 mapped with timezone_id "Europe/Paris"
found: C:\Users\mabenamara\AppData\Local\WebGrab+Plus\siteini.pack\Germany\openwebif.ini -- Revision 00

!! SiteIni openwebif.ini contains active debug arguments !!!
!! Expect slower processing!!

processing C:\Users\mabenamara\AppData\Local\WebGrab+Plus\test.xml ...
Found existing channel (xmltv_id=Eurosport 2 HD Xtra) in the config file
Found existing channel (xmltv_id=Fox HD) in the config file
....

      i=index  .=same  c=change  g=gab  r=replace  n=new

Group (0) :
update requested for - 2 - out of - 2 - channels for 6 day(s)
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Eurosport 2 HD Xtra) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Eurosport 2 HD Xtra
Generic syntax exception:
   message:
no index page data received from Eurosport 2 HD Xtra
unable to update channel, try again later
Existing guide data restored!
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Fox HD) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Fox HD
Generic syntax exception:
   message:
no index page data received from Fox HD
unable to update channel, try again later
Existing guide data restored!

Job finished at 11/02/2018 14:41:19 done in 0s

sure i did test, thats why i modified the ini

thats the original

**------------------------------------------------------------------------------------------------
* @header_start
* WebGrab+Plus ini for grabbing EPG data from TvGuide websites
* @Site: OpenWebIf - Enigma box
* @MinSWversion:
* @Revision 0 - [27/10/2014] Willy De Wilde
*   - creation
* @Remarks:
* @header_end
**------------------------------------------------------------------------------------------------

site {url=openwebif|timezone=UTC+01:00|maxdays=6.1|cultureinfo=en-GB|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="4">2</retry>}

urldate.format {daycounter|0}
url_index{url|http://192.168.1.19/ajax/epgpop?sref=|channel|}

index_showsplit.scrub {multi ()|<table style=||</table>}
index_start.scrub {single (includeblock=2)|<td>||</td>|}
index_stop.scrub {single ()|<td valign="top">||</td>|}
index_title.scrub {single (includeblock=1)|<td>||</td>|}
index_subtitle.scrub {single (includeblock=4)|<td>||</td>|}
index_description.scrub {single ()|<td colspan="2" rowspan="2">||</td>|}

i have openwebif, but /ajax/epgpop.... doesnt respond at all, thats why i changed to that what gives me results

a sample result see attachment

/ajax/epgpop?sref=.... doesnt work here at all

/web/epgservice?sRef=... working in browser

none working in webgrab

tetsing now

now, respond is

Job started at 11/02/2018 17:18:33
timezone=UTC+01:00 mapped with timezone_id "Europe/Paris"
found: C:\Users\mabenamara\AppData\Local\WebGrab+Plus\siteini.pack\Germany\openwebif.ini -- Revision 00

!! SiteIni openwebif.ini contains active debug arguments !!!
!! Expect slower processing!!

processing C:\Users\mabenamara\AppData\Local\WebGrab+Plus\test.xml ...
Found existing channel (xmltv_id=Eurosport 2 HD Xtra) in the config file
Found existing channel (xmltv_id=Fox HD) in the config file
....

      i=index  .=same  c=change  g=gab  r=replace  n=new

Group (0) :
update requested for - 2 - out of - 2 - channels for 6 day(s)
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Eurosport 2 HD Xtra) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Eurosport 2 HD Xtra
Generic syntax exception:
   message:
no index page data received from Eurosport 2 HD Xtra
unable to update channel, try again later
Existing guide data restored!
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Fox HD) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Fox HD
Generic syntax exception:
   message:
no index page data received from Fox HD
unable to update channel, try again later
Existing guide data restored!

Job finished at 11/02/2018 17:18:34 done in 0s

ini looks like this now

site {url=openwebif|timezone=UTC+01:00|maxdays=6.1|cultureinfo=de-GB|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="4">2</retry>}

urldate.format {daycounter|0}
url_index{url|http://root:password@192.168.1.103/web/epgservice?sRef=|channel|}

index_showsplit.scrub {multi|<e2eventlist>|<e2event>|</e2event>|</e2eventlist>}
index_showsplit.modify {(debug)}
index_start.scrub {single|<e2eventstart>||</e2eventstart>|</e2eventstart>}
index_stop.scrub {single ()|<td valign="top">||</td>|}
index_subtitle.scrub {single (includeblock=4)|<td>||</td>|}
index_duration.scrub {single|<e2eventduration>||</e2eventduration>|</e2eventduration>}
index_title.scrub {single|<e2eventtitle>||</e2eventtitle>|</e2eventtitle>}
index_description.scrub {regex||<e2eventdescription[^>]*>(.*?)</e2eventdescription[^>]*>||}

im using a VU+ 4k, VTI 13, openweb interface

no AV, same mashine as where i am using browser ... so pretty sure its not blocked

also reachable through command line

C:\Users\mabenamara\AppData\Local\WebGrab+Plus>ping 192.168.1.103

Ping wird ausgeführt für 192.168.1.103 mit 32 Bytes Daten:
Antwort von 192.168.1.103: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.1.103: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.1.103: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.1.103: Bytes=32 Zeit<1ms TTL=64

Ping-Statistik für 192.168.1.103:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 0ms, Maximum = 0ms, Mittelwert = 0ms

nothing i can see ...

url_index{url(debug)|http://root:password@192.168.1.103/web/epgservice?sRef=|channel|}

may i ask if |channel| is the right argument ? i ant see anywhere that the siteid is used in any way ?

Job started at 11/02/2018 17:30:56
timezone=UTC+01:00 mapped with timezone_id "Europe/Paris"
found: C:\Users\mabenamara\AppData\Local\WebGrab+Plus\siteini.pack\Germany\openwebif.ini -- Revision 00

!! SiteIni openwebif.ini contains active debug arguments !!!
!! Expect slower processing!!

processing C:\Users\mabenamara\AppData\Local\WebGrab+Plus\test.xml ...
Found existing channel (xmltv_id=Fox HD) in the config file
Found existing channel (xmltv_id=Eurosport 2 HD Xtra) in the config file
....

      i=index  .=same  c=change  g=gab  r=replace  n=new

Group (0) :
update requested for - 2 - out of - 2 - channels for 6 day(s)
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Fox HD) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Fox HD
Generic syntax exception:
   message:
no index page data received from Fox HD
unable to update channel, try again later
Existing guide data restored!
(   1/2   ) OPENWEBIF -- chan. (xmltv_id=Eurosport 2 HD Xtra) -- mode Force

no robots data found
skipped robots check
i
Unable to update channel Eurosport 2 HD Xtra
Generic syntax exception:
   message:
no index page data received from Eurosport 2 HD Xtra
unable to update channel, try again later
Existing guide data restored!

Job finished at 11/02/2018 17:30:57 done in 0s

i already checked that, sadly no other one here ...

<channel xmltv_id="Fox HD" site_id="1:0:19:7C:6:85:FFFF0000:0:0:0:" site="openwebif" update="i">Fox HD</channel>
<channel xmltv_id="Eurosport 2 HD Xtra" site_id="1:0:19:30D7:413:1:C00000:0:0:0:" site="openwebif" update="i">Eurosport 2 HD Xtra</channel>

here it is

thanks again for pointing to an connection ;)

now, when i look at the results, doesnt look to good to me in terms of usability for tvheadend, when i check the inputs, can i return and may ask how to modify so it fits better ?

1st, there are alot missing in results, example, when i manually check via browser

<e2event>

<e2eventid>4474</e2eventid>

<e2eventstart>1518381000</e2eventstart>

<e2eventduration>3600</e2eventduration>

<e2eventcurrenttime>1518375209</e2eventcurrenttime>

<e2eventtitle>Fußball</e2eventtitle>

<e2eventdescription>Matchday Highlights</e2eventdescription>

<e2eventdescriptionextended>Alle Spiele des aktuellen Bundesliga-Spieltags</e2eventdescriptionextended>

<e2eventservicereference>1:0:19:30D7:413:1:C00000:0:0:0:</e2eventservicereference>

<e2eventservicename>Eurosport 2 HD Xtra</e2eventservicename>

</e2event>

<e2event>

<e2eventid>4475</e2eventid>

<e2eventstart>1518384600</e2eventstart>

<e2eventduration>6300</e2eventduration>

<e2eventcurrenttime>1518375209</e2eventcurrenttime>

<e2eventtitle>Fußball</e2eventtitle>

<e2eventdescription>Matchday</e2eventdescription>

<e2eventdescriptionextended>Bundesliga, 22. Spieltag: RB Leipzig - FC Augsburg</e2eventdescriptionextended>

<e2eventservicereference>1:0:19:30D7:413:1:C00000:0:0:0:</e2eventservicereference>

<e2eventservicename>Eurosport 2 HD Xtra</e2eventservicename>

</e2event>

<e2event>

<e2eventid>4476</e2eventid>

<e2eventstart>1518390900</e2eventstart>

<e2eventduration>1800</e2eventduration>

<e2eventcurrenttime>1518375209</e2eventcurrenttime>

<e2eventtitle>Fußball</e2eventtitle>

<e2eventdescription>Matchday</e2eventdescription>

<e2eventdescriptionextended>Analyse des Spiels RB Leipzig - FC Augsburg</e2eventdescriptionextended>

<e2eventservicereference>1:0:19:30D7:413:1:C00000:0:0:0:</e2eventservicereference>

<e2eventservicename>Eurosport 2 HD Xtra</e2eventservicename>

</e2event>

<e2event>

<e2eventid>4477</e2eventid>

<e2eventstart>1518392700</e2eventstart>

<e2eventduration>3600</e2eventduration>

<e2eventcurrenttime>1518375209</e2eventcurrenttime>

<e2eventtitle>Fußball</e2eventtitle>

<e2eventdescription>Matchday Highlights</e2eventdescription>

<e2eventdescriptionextended>Alle Spiele des aktuellen Bundesliga-Spieltags</e2eventdescriptionextended>

<e2eventservicereference>1:0:19:30D7:413:1:C00000:0:0:0:</e2eventservicereference>

<e2eventservicename>Eurosport 2 HD Xtra</e2eventservicename>

</e2event>

<e2event>

<e2eventid>4478</e2eventid>

<e2eventstart>1518396300</e2eventstart>

<e2eventduration>4500</e2eventduration>

<e2eventcurrenttime>1518375209</e2eventcurrenttime>

<e2eventtitle>Eishockey</e2eventtitle>

<e2eventdescription>Eishockey: Frauen, Vorrunde</e2eventdescription>

<e2eventdescriptionextended>Kanada - Olympic Athlete from Russia</e2eventdescriptionextended>

<e2eventservicereference>1:0:19:30D7:413:1:C00000:0:0:0:</e2eventservicereference>

<e2eventservicename>Eurosport 2 HD Xtra</e2eventservicename>

</e2event>

and result in xml there are missing events

    <desc lang="en">Matchday Highlights. Alle Spiele des aktuellen Bundesliga-Spieltags</desc>
  </programme>
  <programme start="20180211213000 +0100" stop="20180212043000 +0100" channel="Eurosport 2 HD Xtra">
    <title lang="en">Fußball</title>
    <desc lang="en">Matchday. Bundesliga, 22. Spieltag: RB Leipzig - FC Augsburg</desc>
  </programme>
  <programme start="20180212043000 +0100" stop="20180212073000 +0100" channel="Eurosport 2 HD Xtra">
    <title lang="en">Eishockey</title>
    <desc lang="en">Eishockey: Frauen, Vorrunde. Kanada - Olympic Athlete from Russia</desc>
  </programme>
  <programme start="20180212073000 +0100" stop="20180212120000 +0100" channel="Eurosport 2 HD Xtra">
    <title lang="en">Eishockey</title>
    <desc lang="en">Eishockey: Frauen, Vorrunde. Japan - Schweiz</desc>

log attached, as all events are catched but not written to xml

i see its overlapping times ... strange, ahh well, enough for today ;) thanks again, i ll test more tom morning.

log attached just in case u see something, also the xml return from browser, may easier to check what happens.

Hi, looking ok now for my missing channels data here, majors i grab from german site cause for example the series and episode infos are not grabbed with the commands enigma2 offers, roughly the vu box has them all nice added ;) nevermind.

what would interest me, EPG length, the grabber doesnt add all data, cutting off affter 2 days, even the settings say 6 days ...

may an idea ?

thank again ;)

openwebinterface ?

ok, i extended it a little when editing the .ini

url_index{url|http://192.168.1.103/web/epgservice?sRef=|channel|&time=14400}

like this the web is grabbing further, i guess its not using the "days" settings in .ini

thanks for pointing direction ;)

understood, thanks for the explanation.

and yes, its minutes, so 10 days i just added there, even there aint that many available.

Why did i add this setting, example, a channel like eurosport 2 hd extra returns without my addition 61 entries (within 3 days),

when i add time=14400 it returns more 72 ... so i get some extra epg data in return, even its all lower then the 6 days ...

thanks again, im good now.

sorry to ask again, i see my impoted epg from openwebif has a offset for 1 hour.

even the timezones are setted same as the other ini i use (vodafone).

site {url=vodafone.de|timezone=Europe/Berlin|

site {url=openwebif|timezone=Europe/Berlin|m

is this the setting to change the exported time ? so i could just adjust here ? forexample, change to UTC+2 even im on UTC+1 ?

here is a example from guide.xml, the 1st one is from vodafone and correct, 2nd and third are from openenigma and offset by 1 hour in TVHeadend

  <programme start="20180216150000 +0100" stop="20180216163500 +0100" channel="Sky Cinema HD">
    <title lang="de">The Boss Baby</title>

  <programme start="20180215201500 +0100" stop="20180215215000 +0100" channel="Sky Cinema +1 HD">
    <title lang="en">The Boss Baby</title>

  <programme start="20180216191500 +0100" stop="20180216205000 +0100" channel="Sky Cinema +24 HD">
    <title lang="en">The Boss Baby</title>

Config is as described above, all on Europ/Berlin. checked all server settings, TVH, Webgrab+,VU+ box, all showing correct time on "date",

thats why i think i have to adjust manually even if its "wrong" then.

here we go and now it fits, thanks again

my .ini (change IP to fit your enigma box)

site {url=openwebif|timezone=UTC|maxdays=6.1|cultureinfo=en-GB|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="4">2</retry>}

urldate.format {daycounter|0}
url_index{url|http://192.168.1.103/web/epgservice?sRef=|channel|&time=14400}
index_showsplit.scrub {multi|<e2eventlist>|<e2event>|</e2event>|</e2eventlist>}
index_start.scrub {single|<e2eventstart>||</e2eventstart>|</e2eventstart>}
index_duration.scrub {single|<e2eventduration>||</e2eventduration>|</e2eventduration>}
index_duration.modify {calculate(format=F0)|60 /}
index_title.scrub {single|<e2eventtitle>||</e2eventtitle>|</e2eventtitle>}
index_subtitle.scrub {single|<e2eventdescription>||</e2eventdescription>|</e2eventdescription>}
index_description.scrub {single|<e2eventdescriptionextended>||</e2eventdescriptionextended>|</e2eventdescriptionextended>}

a sample line from xml

<channel xmltv_id="Eurosport 2 HD Xtra" site_id="1:0:19:30D7:413:1:C00000:0:0:0:" site="openwebif" update="f">Eurosport 2 HD Xtra</channel>