Loading…
Tilbage

Profilbillede

Replication & GeoJSON

Dmytro Gokun

Hi all,

 

first of all, I'm sorry for asking in English, but unfortunatelly I cannot speak Danish. So, please bear with me :)

I work for a Danish company and we are trying to set up replication as described here: http://dawa.aws.dk/dok/guide/replikering

It all works well, but there is only one problem: it does not suport GeoJSON format. And that's exactly what we need for our use case: geographic information. For example, for roads, we need MultiLineStrings which represent their location. That information is returned when extracting all roads using http://dawa.aws.dk/vejstykker?format=geojson , but it is not when using replication. Same for postal codes and other entities, GeoJSON is not supported.

So, i'm a bit puzzled here. Looks like the only options we have is to download the whole database every time we want fresh data. That is exactly what replication process is build to avoid to.

Does anybody have an idea how to help this? Perhaps I am missing something very obvious? Any help is appreciated.

Thanks in advance, 

Dmytro

Hi Dmytro

Replication of roads and other objects geometry is not supported. The wish is on our roadmap, but we do not have any concrete plans for when it will be implemented.
Do you have the opportunity to describe your needs for geometry data?

Sincerely.
Finn

Hi Finn,

 

basically we want to implement reverse geocoding for roads.

We do not want to use DAWA API for that because:

1) we will need to do multiple calls per second (and we are not sure if DAWA API's terms of use allow us to do that. Does it?);

2) we want high availability for this service, so we do not want to introduce a dependency to a 3-rd party APIs/servers.

3) 1 minor problem with DAWA's road reverse geocoding: it would not return the distance to the road found, so it's hard to estimate the quality of that find - should we use the results or should we abandon them?

 

So, in order to implement this reverse geocoding ourselves, we need this geometry data (right now, we need that for roads & postal codes, but this may expand in the future). Without GeoJson support in replication API, the only choice we have is to periodically download the whole datasets. Again, we are not sure if that will violate terms of use if we do that, say, once a week?

Dmytro.

 

Hi Dmytro,

You are allowed to make as many calls to DAWA as you need. We employ sophisticated traffic shaping to ensure that any single client IP cannot overload our system. If clients attempt to consume more data than we can provide, the server resources are divided equally among the clients. If a single client consume too many resources, however, that client may experience longer latencies, slower download speeds and (eventually) timeouts in order to protect QoS for other clients.

In addition, we have an auto-scaling environment, so additional servers are automatically added as needed.

We only block clients from accessing DAWA, if the traffic pattern is the result of an incorrectly implemented client, and this happens extremely rarely.

However, we currently have a hard limit of 100 requests per second for any single client IP.

Feel free to download the entire dataset on a daily or weekly basis if that is what you need. Making 5-10 requests per second to DAWA is also no problem at all.

I definitely understand why you do not want to rely on third party services. However, DAWA is never taken down for maintenance, and we have succeeded in maintaining a very high uptime exceeding 99.97% during the last couple of years of operation. 

Hi Anders,

 

thanks a lot for yor detailed and comprehansive answer! That's exactly what we needed to know (Y).

 

I think, we'll start by calling your API directly and then we will switch to a local replica when our project grows. If, by that time, replication API supports geometry data - that would be great, as we can then minimize the traffic.

 

Thanks a lot,

Dmytro

ændret af Dmytro Gokun (26.03.2018)