navit-project.org

With bigger Openstreetmap-Files there is a problem with nodes. Maptool cannot handle all nodes.

But p.e. the alps-region is to big and so a lot of nodesare missing. Maptool shows hints, that it cannot use these nodes. Parts of ways then missing on the map. Its diverted over the map, so you have to search for a while, to find missing nodes.

I used resolved Multipolygons, so that I need 10% more nodes than without.

I started maptool with -S 600000000, with 600 MB Memory to use. Could this be a problem? That not so much nodes given with less memory?

As for now, maptool is able to process whole planet osm file without any problem, we do it every day on our map server. Maptool uses unsigned 32bit integer to store node id in some places, but current osm biggest number data is more than 30% below our limit.


Can you qoute the exact phrase which maptool uses to give you a hint? Attaching the whole maptool log might be a good idea too.
Can you give us some node ids which exist in osm database but is not present in extract of corresponding area downloaded from maps.navit-project.org?


I have no exact knowledge what and how your program does. Why does it produce additional nodes when resolving multipolygons?


With that option, maptool will use 600Mb of memory for active slice and some additional memory to store other data. I think that additional space would be around 1 gigabyte for planet.bin.
The less is slice size, the more time it would take to process map. If you'll run out of physical memory, performance will be affected due to swapping. If your system has not enough memory resources (physical memory + swap), maptool will crash.
I expect no other effects to be produced by using different -S values.

I dont now, how to make a log-file. I added a picture with the maptool-warnings and a picture of the map with a missing part of a street.

I tested without resolved Multipolygones and with resolved Multipolygones at the end and start of way-section. The result is, that the problem is always there, but more parts of streets missing with resolved Multipolygones. If they are at the beginning of the way-section, they are complete, at the end, no resolved appears on the map. That means, that the ways at the end of the way-sections are the problem.

The number of nodes is always the same. I added no nodes with resolved Multipolygones, only ways. But with resolved Mutlipolygones, more nodes are used and maptool test the reference to the nodes.

Different -S size make no effort, maptool crashes only with more than 900MB on my computer. Swap is not used, the 2 GB of my computer seems enough. I have a 32-Bit-System.

I looked for the reason myself and cannot find it. I saw a lot of type mismatch in osm.c. unsigned long long ids matched with long long ids.

g_Hashtable has only unsigned int Keys maximal, but there is often in attr_hash a casting to long. Variable int idx get a long from g_Hashtable. But this would work, because the long-values is never bigger than int here. The rule-file has only a few elements, much less than unsigned int, so there is no danger, only a bit strange.

In line 1250, there is used realloc. I think, there should be g_realloc. In the whole file g_realloc and similiar is used. But this is not very important.

In line 1235 attr_present[idx]=2. I think, it should be 3 there. But this never used in rule-file-part, so there should be go nothing wrong yet.

The only thing, that I could imagine is, that it works on 64-bit-Systems, because long int, long long and only int has different number of bytes on different systems. In C, variables are not platform-independent.

maptool says, that there are 134.000.000 nodes. This should be not to much for unsigned long.

I was talking about some extracts from the file produced with

Anyway, your example was quite helpful.

I was thinking that the problem is node id values. Ways at the end of way section (whith higher way ids) in average would have higher node numbers and more probably would cross any existing limitations.
But the missing way from your sample screen unfortunately doesnt seem to have any node id which might cross the signed 32 bit int limit.

The only problem I've noticed is that maptool sometimes still uses signed int formats to report node ids. It's quite confusing, so I'll fix it.


This is done intentionally. We're counting weight to find matches which use less wildcards than others. So "%s=*" and "*=%s" are given the same values.


I have tired to reproduce this on my 32 bit system, but had no success. Probably it was because I was not using big files with loads of fresh data. But can't you you be getting your errors because you have a broken dataset? Can you try to reload it from the source?


sure.

Actually, maptool does not assign any internal numbering to all nodes. It uses osm ids when it has to reference a node. These ids are currently expected to be cut down to 32bit unsigned ints.

The problems begins with alps-region from Geofabrik. Baden-Wuerttemberg, part of germany was still ok. No such warning occcured.

And if resolved Multipolygons at the end of waysection disapear, and at the beginning of way-section the are ok, so it seems to me a problem at the end. A unexpected limitation of ways or number of used nodes-references.
Ok, and I will download now the region new and make the warning-report

I can confirm now that alps-latest.osm.pbf from geofabric produces on 32bit system messages like


There are no such messages produced by maptool on 64 bit system.

This can also be some bug in our pbf processing routines, not directly in osm.c.

I tried the new alps-file and it was the same. The new file is protobuf-file and the old file was bzip2-compressed and after decompressing used with maptool.

The log-file was empty, dont know why.

Thanks a lot for your testing on 32/64 bit!

Because number of nodes is not at long-limit and with number of ways the same, perhaps there are still very small memoryleaks, and they appear only with less memory and big input-files. I saw a lot of uncontrolled strcpy and similiar in osm.c and tile.c. It seemed to be sure, buffers were very big, but perhaps?

There was a missing & char in my command line sample. I have fixed it now. If you have followed that wrong sample, your log is in file named 1.

Ah, ok! Ihave found it now! I deleted a lot of warnings, the logfile is originally 110MB, full with the same warning. There are 3 Points in a line, where I deleted Warnings.

i find a difference between 32bit and 64bit machines. longlong both 64bit, but
long and int are 32bit on 32bit machine and long,int is 64 bit on 64-bit-machine. So long-key in g_Hashtable is in fact long long int.

The nodes-id is since 2013 above 32 bit. Perhaps not the number of nodes, because there could be some gaps in it.

I tested to change in osm.c all strcpy to strncpy a.s.o., this made no difference in maptool-warnings.

I tried to change all id to type insigned long long, but then the same warnings come earlier und with smaller files. It seems, that maptool needs the type-mismatch, to realize the integer-limit, if values are negative. There are a lot of workaround to the limits of g_HashTable.

Actual problem was in 32bit file positioninig interface we were using in maptool on 32bit machines. So maptool was failing to advance a temporary file above 2GiB. I have switched io interface to 64bit and it should be working fine since r5745.