If you are using monolog (https://github.com/Seldaek/monolog) (or some custom logging program) to create logs, what is a good way to analyze them if you have tens to hundreds of thousands of events per day? I am not sure where to start looking for an industrial strength log parser. It would be helpful analyzer could also handle php logs, apache logs, and other common log types.
My original plan was: Log every event to a mysql table (or tables), then run the appropriate SQL queries on them when needed, and also have a custom-made admin interface for displaying certain things, like site logins in the last N minutes. Cron jobs would provide updated stats for certain queries on the logs or shoot out emails for some events.
But I read that using a mysql database for logging of this magnitude is a bad idea. But this takes away SQL, which is what I was going to rely on to make any sense of the logs at all without writing a custom parser of text files.
Is there a standard solution to this problem? I did some googling and I couldn’t narrow down what I am looking for – logging analyzer sites are somewhat opaque and full of jargon. I came across things like GrayLog2, ElasticSearch, LogStasch, Hadoop, Sentry, and some others, but I really don’t know if these are even in the realm of what I am looking for. What ways can I proceed from here?
Hi,
The way you compile the log, a.k.a. the logging program, should you reverse the other way around. If you log something in a certain method, then just write a code that will decode the way it was written, so it can be humanly read.
I suggest you use text-files if it’s on a large scale, since mysql can be hard to handle with large data. Try looking for a logging system that has a parser as well.(or write one by yourself.). Other then that, I don’t see another way around this.
I can do this all by myself, but I am guessing there is a pre-built solution out there to this issue since it is likely not uncommon. I also don’t have any idea of best practices in this area, so starting from scratch will mean rewriting it later. (I am also not sure if php is the best way to pursue this, other than making the logs themselves.)
My problem is just that I dont know where to start in this area, and don’t know the terrain. What are people generally using to parse log files?