luya — 2013-09-07T06:37:04-04:00 — #1
I have a website running on App Engine and I store my data in Datastore. If the text contains Turkish letters, they are replaced with ? in a diamond. But interestingly, sometimes I edit them and they appear fine or while they appear correct they start to contain <?> again.
I don't know when it happens and when it doesn't. So I change something in the code, then it works, but later I see those <?> again. I call it mysterious because of that and I don't know how to explain better. I will try again if it's not clear.
It is 2013 and I am struggling with characters :/ By the way, I am not expert on web development, I am learning while making this. So maybe I forgot a very simple thing.
Please create a record here to see (I can delete them later or you can delete afterwards)
These are the turkish characters for copy pasting: ş ç ı ğ ü ö
luya — 2013-09-08T18:09:56-04:00 — #2
I sort of re generate the error.
If I edit all the fields in the form leaving no strange char of <?>, the turkish letters sent to server are replaced to <?>
but if I leave one <?> in the form and re-write the rest, then it saves what is seen exactly.
looks like my browser or something tries to guess the encoding and converts into another encoding.
I check in the server right after parsing json, and they are already replaced.
Actually I should check also before json.Unmarshal. I'll try now
luya — 2013-09-08T18:34:30-04:00 — #3
Still solutionless. I compared the request headers in inspector window for two form submissions and they are not much different. how can I debug more?
luya — 2013-09-09T16:21:13-04:00 — #4
My silly work around is to send a <?> which is String.fromCharCode(65533) with the form data.
It is also interesting that if I type "ş ç ı ğ ü ö" it worked fine but when I write a meaningful sentence it guess the context (my opinion). So I didn't understand when it happens or not.