navit-project.org

forum for navit navigation tool
It is currently 16 Aug 2017, 15:11
View unanswered posts | View active topics


All times are UTC


Forum rules


Feel free to ask anything here related to the development process - coding, creating new features, fixing bugs and custom changes of Navit.

Note: For reporting bugs, use the bug tracker.



Post new topic Reply to topic  [ 11 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Id-Limitation
PostPosted: 09 Dec 2013, 13:13 
Offline

Joined: 26 Sep 2013, 10:59
Posts: 50
With bigger Openstreetmap-Files there is a problem with nodes. Maptool cannot handle all nodes.

But p.e. the alps-region is to big and so a lot of nodesare missing. Maptool shows hints, that it cannot use these nodes. Parts of ways then missing on the map. Its diverted over the map, so you have to search for a while, to find missing nodes.

I used resolved Multipolygons, so that I need 10% more nodes than without.

I started maptool with -S 600000000, with 600 MB Memory to use. Could this be a problem? That not so much nodes given with less memory?


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 11 Dec 2013, 19:03 
Offline

Joined: 09 Jul 2013, 17:41
Posts: 82
Mapmapper wrote:
With bigger Openstreetmap-Files there is a problem with nodes. Maptool cannot handle all nodes.

As for now, maptool is able to process whole planet osm file without any problem, we do it every day on our map server. Maptool uses unsigned 32bit integer to store node id in some places, but current osm biggest number data is more than 30% below our limit.

Mapmapper wrote:
But p.e. the alps-region is to big and so a lot of nodesare missing. Maptool shows hints, that it cannot use these nodes. Parts of ways then missing on the map. Its diverted over the map, so you have to search for a while, to find missing nodes.

Can you qoute the exact phrase which maptool uses to give you a hint? Attaching the whole maptool log might be a good idea too.
Can you give us some node ids which exist in osm database but is not present in extract of corresponding area downloaded from maps.navit-project.org?

Mapmapper wrote:
I used resolved Multipolygons, so that I need 10% more nodes than without.

I have no exact knowledge what and how your program does. Why does it produce additional nodes when resolving multipolygons?

Mapmapper wrote:
I started maptool with -S 600000000, with 600 MB Memory to use. Could this be a problem? That not so much nodes given with less memory?

With that option, maptool will use 600Mb of memory for active slice and some additional memory to store other data. I think that additional space would be around 1 gigabyte for planet.bin.
The less is slice size, the more time it would take to process map. If you'll run out of physical memory, performance will be affected due to swapping. If your system has not enough memory resources (physical memory + swap), maptool will crash.
I expect no other effects to be produced by using different -S values.


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 13 Dec 2013, 00:44 
Offline

Joined: 26 Sep 2013, 10:59
Posts: 50
I dont now, how to make a log-file. I added a picture with the maptool-warnings and a picture of the map with a missing part of a street.

I tested without resolved Multipolygones and with resolved Multipolygones at the end and start of way-section. The result is, that the problem is always there, but more parts of streets missing with resolved Multipolygones. If they are at the beginning of the way-section, they are complete, at the end, no resolved appears on the map. That means, that the ways at the end of the way-sections are the problem.

The number of nodes is always the same. I added no nodes with resolved Multipolygones, only ways. But with resolved Mutlipolygones, more nodes are used and maptool test the reference to the nodes.

Different -S size make no effort, maptool crashes only with more than 900MB on my computer. Swap is not used, the 2 GB of my computer seems enough. I have a 32-Bit-System.

I looked for the reason myself and cannot find it. I saw a lot of type mismatch in osm.c. unsigned long long ids matched with long long ids.

g_Hashtable has only unsigned int Keys maximal, but there is often in attr_hash a casting to long. Variable int idx get a long from g_Hashtable. But this would work, because the long-values is never bigger than int here. The rule-file has only a few elements, much less than unsigned int, so there is no danger, only a bit strange.

In line 1250, there is used realloc. I think, there should be g_realloc. In the whole file g_realloc and similiar is used. But this is not very important.

In line 1235 attr_present[idx]=2. I think, it should be 3 there. But this never used in rule-file-part, so there should be go nothing wrong yet.

The only thing, that I could imagine is, that it works on 64-bit-Systems, because long int, long long and only int has different number of bytes on different systems. In C, variables are not platform-independent.

maptool says, that there are 134.000.000 nodes. This should be not to much for unsigned long.

Attachment:
Bildschirmfoto1.jpg
Bildschirmfoto1.jpg [ 86.46 KiB | Viewed 5635 times ]

Attachment:
Bildschirmfoto2.jpg
Bildschirmfoto2.jpg [ 40.18 KiB | Viewed 5635 times ]


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 13 Dec 2013, 20:39 
Offline

Joined: 09 Jul 2013, 17:41
Posts: 82
Mapmapper wrote:
I dont now, how to make a log-file. I added a picture with the maptool-warnings and a picture of the map with a missing part of a street.

I was talking about some extracts from the file produced with
Code:
maptool -i in.osm out.bin 2>&1 |tee log.txt

Anyway, your example was quite helpful.
Mapmapper wrote:
I tested without resolved Multipolygones and with resolved Multipolygones at the end and start of way-section. The result is, that the problem is always there, but more parts of streets missing with resolved Multipolygones. If they are at the beginning of the way-section, they are complete, at the end, no resolved appears on the map. That means, that the ways at the end of the way-sections are the problem.

I was thinking that the problem is node id values. Ways at the end of way section (whith higher way ids) in average would have higher node numbers and more probably would cross any existing limitations.
But the missing way from your sample screen unfortunately doesnt seem to have any node id which might cross the signed 32 bit int limit.
Mapmapper wrote:
...
I looked for the reason myself and cannot find it. I saw a lot of type mismatch in osm.c. unsigned long long ids matched with long long ids.
...
In line 1250, there is used realloc. I think, there should be g_realloc. In the whole file g_realloc and similiar is used. But this is not very important.

The only problem I've noticed is that maptool sometimes still uses signed int formats to report node ids. It's quite confusing, so I'll fix it.

Mapmapper wrote:
In line 1235 attr_present[idx]=2. I think, it should be 3 there. But this never used in rule-file-part, so there should be go nothing wrong yet.

This is done intentionally. We're counting weight to find matches which use less wildcards than others. So "%s=*" and "*=%s" are given the same values.

Mapmapper wrote:
The only thing, that I could imagine is, that it works on 64-bit-Systems, because long int, long long and only int has different number of bytes on different systems. In C, variables are not platform-independent.

I have tired to reproduce this on my 32 bit system, but had no success. Probably it was because I was not using big files with loads of fresh data. But can't you you be getting your errors because you have a broken dataset? Can you try to reload it from the source?

Mapmapper wrote:
maptool says, that there are 134.000.000 nodes. This should be not to much for unsigned long.

sure.

Actually, maptool does not assign any internal numbering to all nodes. It uses osm ids when it has to reference a node. These ids are currently expected to be cut down to 32bit unsigned ints.


Last edited by tryagain on 14 Dec 2013, 09:31, edited 1 time in total.

 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 14 Dec 2013, 00:10 
Offline

Joined: 26 Sep 2013, 10:59
Posts: 50
The problems begins with alps-region from Geofabrik. Baden-Wuerttemberg, part of germany was still ok. No such warning occcured.

And if resolved Multipolygons at the end of waysection disapear, and at the beginning of way-section the are ok, so it seems to me a problem at the end. A unexpected limitation of ways or number of used nodes-references.
Ok, and I will download now the region new and make the warning-report


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 14 Dec 2013, 10:26 
Offline

Joined: 09 Jul 2013, 17:41
Posts: 82
I can confirm now that alps-latest.osm.pbf from geofabric produces on 32bit system messages like
Code:
OSM Warning:http://www.openstreetmap.org/browse/way/111804279 Non-existing reference to http://www.openstreetmap.org/browse/node/-1777237200
OSM Warning:http://www.openstreetmap.org/browse/way/111804349 Non-existing reference to http://www.openstreetmap.org/browse/node/-1750026376


There are no such messages produced by maptool on 64 bit system.

This can also be some bug in our pbf processing routines, not directly in osm.c.


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 14 Dec 2013, 13:17 
Offline

Joined: 26 Sep 2013, 10:59
Posts: 50
I tried the new alps-file and it was the same. The new file is protobuf-file and the old file was bzip2-compressed and after decompressing used with maptool.

The log-file was empty, dont know why.

Thanks a lot for your testing on 32/64 bit!

Because number of nodes is not at long-limit and with number of ways the same, perhaps there are still very small memoryleaks, and they appear only with less memory and big input-files. I saw a lot of uncontrolled strcpy and similiar in osm.c and tile.c. It seemed to be sure, buffers were very big, but perhaps?


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 14 Dec 2013, 13:53 
Offline

Joined: 09 Jul 2013, 17:41
Posts: 82
Mapmapper wrote:
The log-file was empty, dont know why.


There was a missing & char in my command line sample. I have fixed it now. If you have followed that wrong sample, your log is in file named 1.


 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 14 Dec 2013, 17:11 
Offline

Joined: 26 Sep 2013, 10:59
Posts: 50
Ah, ok! Ihave found it now! I deleted a lot of warnings, the logfile is originally 110MB, full with the same warning. There are 3 Points in a line, where I deleted Warnings.

i find a difference between 32bit and 64bit machines. longlong both 64bit, but
long and int are 32bit on 32bit machine and long,int is 64 bit on 64-bit-machine. So long-key in g_Hashtable is in fact long long int.

The nodes-id is since 2013 above 32 bit. Perhaps not the number of nodes, because there could be some gaps in it.

I tested to change in osm.c all strcpy to strncpy a.s.o., this made no difference in maptool-warnings.

I tried to change all id to type insigned long long, but then the same warnings come earlier und with smaller files. It seems, that maptool needs the type-mismatch, to realize the integer-limit, if values are negative. There are a lot of workaround to the limits of g_HashTable.


Attachments:
logfile.txt.tar.gz [13.77 KiB]
Downloaded 172 times
 Profile  
 
 Post subject: Re: Id-Limitation
PostPosted: 22 Feb 2014, 20:06 
Offline

Joined: 09 Jul 2013, 17:41
Posts: 82
Mapmapper wrote:
I tried to change all id to type insigned long long, but then the same warnings come earlier und with smaller files. It seems, that maptool needs the type-mismatch, to realize the integer-limit, if values are negative. There are a lot of workaround to the limits of g_HashTable.

Actual problem was in 32bit file positioninig interface we were using in maptool on 32bit machines. So maptool was failing to advance a temporary file above 2GiB. I have switched io interface to 64bit and it should be working fine since r5745.


 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ]  Go to page 1, 2  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Silver Orange 2.0.6 for IPB Designed by Skins and Hosting
Converted for phpBB3, based on Royal Blue template by BigB © 2007 2008 AEON KINGS