Login

**esilverm** · 03-29-2019, 11:52 AM

Introduction

As some of you may know, I am the guy who made the SHL Scraper bot that you may find in your team Discord. Back in Season 38, I hated having to search for the index and then specifically search for my player, so I devised a way to do it without even visiting the site. I mean, if you wanted to see specifics you would have to visit it, but if you just wanted to see your player’s point totals and some other important stats, my bot was what you needed. Truth be told, my bot was pretty trash. I wrote it over the course of a day or two, and it used really shitty techniques to gather data. What it did was download an XML file from the SMJHL index which contained data about players, convert it to JSON, and then synchronously add objects containing player data from the SHL as it didn’t have an XML file. All these writing commands put a lot of stress on the hosting service I was using for my bot, so it had trouble staying online.

A few seasons later the XML file I had been using to pull data from was removed from the index. I told everyone that I was going to fix it, but then I just kinda left the SHL because I had a lot going on. So, for the course of several seasons, 18 SHL related servers had a bot that didn’t work, probably in a channel that was left dead for months. Over the course of my break I would check in on the discord servers I was a part of and see people trying to access data and people responding with, “it has been dead for a while.” Honestly, I felt bad, but school came first so I couldn’t help y’all out.

About two weeks ago I was sitting around with nothing to do, and I decided, how about I remake the bot, and then I did. From the ground up, I built the bot you may or may not know is back up and running on your servers. With the new bot, came new techniques for data management, data lookup, and MUCH better layout. Though the bot may not have all the things it had before, it is the same bot you all know and hopefully appreciate. Here I will talk about the new things I have included, how the bot functions (if you didn’t know already), and what is to come in the future.

How I Rewrote It

Note: this will get pretty mathy and if you don’t want to read it just skip ahead

So, I decided that the old way of storing data was really rough and honestly was the worst thing I could’ve done in terms of data storage. For this new iteration of the bot, I am using a data structure called a Hash Table. For those who haven’t taken a class on Data Structures or may have simply forgot about it, a Hash Table is basically a list of n slots. Each slot is assigned a number from 0 (because programming is like that) to or max number n. You can put objects into slots based on a certain key. For the bot, the key for these slots I am using names of players. What happens, is that to transform the player’s name into a number, I get each character’s ascii value and add it up into a sum. This is called our hash. It is basically a numerical value which represents our player’s name. What happens next is we find the modulo (or remainder) of our hash and n so we can find the correct slot to place the player in. Once we have this, we get the slot and add the player. You may be wondering what happens if multiple players end up in the same slot? To fix this problem, I use a technique called separate chaining. What happens is we make the slot a bit bigger to fit the next player and so on. As the number of players in a table grows bigger, so do the slots. Mathematically speaking, once we have n+1 players in our table, we know with certainty that there is a slot with more than one player, so take a number much larger than that and we will have 5-7 players per slot.

Storing the players is the hard part. The easy part is getting the players from the table. When a user requests a player’s data from the bot, the bot hashes the name given and uses that hash to get the player’s data. It is like a mailbox system except you only want one piece of mail from your mailbox and you open it and put it back in.

Rather than gathering the data synchronously (which takes longer), I use asynchronous commands. It literally takes half a second to collect all the data from both leagues with this approach compared to the previous one which took about a minute. If that isn’t an upgrade, I don’t know what is.

Finally, when you type in a name with a special symbol like ø or when you accidentally misspell your player’s name by 2 characters the bot will handle small typos for you! The bot implements a fuzzy string matching algorithm library called fuzzyset.js. It uses an algorithm called Levenshtein distance to find the closest key to what you had entered. I think it is pretty great. In order to make sure people don’t get false positives when searching for their player (for instance the bot saying that Michael Fox = Michael Scarn), I made it so the strings have to be have 80% similarity or above!

Basically the current iteration of the bot is implemented A LOT better than before.

How to Use It

For those who are new to using my bot, there are currently only 3 commands that work:
Player search, store, and random. Here I will include screenshots of the bot in use, because it is tailored to be user-friendly.

Searching for a player

The commands for player search are simple: !player or !p. Both of these commands do the same thing, the shorter one is for faster typing. These commands usually* will require you to include the name of the player you would like to search. You should type your command like this:

[Image: TL1GBG2.png]

or this:

If your search works, you should get a result like this:

[Image: 3mhDw1k.png]

If it doesn’t work, or if the player you are searching for hasn’t played a game yet in the current season or playoffs, it should look like this:

[Image: ubZl5yI.png]

Storing a Player

Sometimes you just want to quickly make a search for your player without typing their names out. This is where the storing command comes in. The command is used like this:

[Image: ARq4jni.png]

If the bot successfully stores your player, it will send this message

[Image: i7MtM6L.png]

If it isn’t successful it will send the same failure message as above. If you successfully store your player, you can simply search for your player’s stats like this:

[Image: pzA305F.png]

Random Player

This is a command that is just for fun. All you have to do is type !r or !random and you will get a random player’s stats from either league.

The Future of the Bot

With the ever-changing state of the league and its sub-tournaments, there will always be more to add. One of the main things that has been requested for me to do is a command that will get IIHF and WJC stats for players. I am intending on adding this before these upcoming tournaments (and if I don’t call me out please). I am interested in building this bot to a certain extent to where everybody can appreciate it for its many uses.

Another idea I have had is the re-implementation of the Team command. Currently, I have the code set up so if I want to add the team command, I can. What is holding me back is the fact that team stats will be different for regular season and playoffs. I will try to implement this before next season.

I am open to any and all suggestions in terms of commands on my bot. Just shoot me a PM and I will be sure to get back to you!

Also, this bot’s code is Open Source (though it may not be the most up to date). Feel free to check it out and judge my quickly put-together implementation.

https://github.com/esilverm/shl-scraper

Thanks for reading this! The link to add the bot is in my sig.

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

**.bojo** · 03-29-2019, 12:35 PM

This is great. If you want to use data from the bojo box I can write some api

**esilverm** · 03-29-2019, 01:01 PM

03-29-2019, 12:35 PM.bojo Wrote: This is great. If you want to use data from the bojo box I can write some api

Honestly, for current season stats it isn't needed, but in the future I could implement a command for previous season stats. So it would be pretty helpful for that

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

*Scrufdaddy* · 03-29-2019, 01:21 PM

This and the recent corsi work are making me itch to get back on my project. Of course, I'll just end up trying to tool my incomplete work into a discord integration instead of actually finishing anything useful Smile

awils13 · 03-29-2019, 01:23 PM

You could add advanced stats if that's something people would like to see (Idk).
I'll be calculating them every season and it's just an excel sheet/csv which you can convert to json and store as a constant variable in the code. Or just read from a file.

Code:
!a Player Name

- Show latest available advanced stats.
Something like that

**.bojo** · 03-29-2019, 01:24 PM

03-29-2019, 01:01 PMesilverm Wrote:
03-29-2019, 12:35 PM.bojo Wrote: This is great. If you want to use data from the bojo box I can write some api

Honestly, for current season stats it isn't needed, but in the future I could implement a command for previous season stats. So it would be pretty helpful for that

I was thinking more for career stats if you go that route.

**esilverm** · 03-29-2019, 01:58 PM

03-29-2019, 01:24 PM.bojo Wrote:
03-29-2019, 01:01 PMesilverm Wrote: Honestly, for current season stats it isn't needed, but in the future I could implement a command for previous season stats. So it would be pretty helpful for that

I was thinking more for career stats if you go that route.

Honestly both could work. I think career stats would be easier to implement but season stats would be neat as well (but likely less useful).

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

**esilverm** · 03-29-2019, 02:00 PM

03-29-2019, 01:23 PMaaronwilson Wrote: You could add advanced stats if that's something people would like to see (Idk).
I'll be calculating them every season and it's just an excel sheet/csv which you can convert to json and store as a constant variable in the code. Or just read from a file.

Code:
!a Player Name

- Show latest available advanced stats.
Something like that

That actually seems pretty interesting. I'll take a look into it! Thanks!

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

*Good_Ole_Kimmy* · 03-29-2019, 03:40 PM

This is absolutely sick man, thanks for doing it

TnlAstatine · 03-29-2019, 03:43 PM

DeletedAtUserRequest · 03-29-2019, 04:14 PM

Welcome back silver!

can it look up team numbers?

|

|

|

|

**esilverm** · 03-29-2019, 05:15 PM

03-29-2019, 04:14 PMMike Izzy Wrote: Welcome back silver!

can it look up team numbers?

Currently, the team function hasn't been implemented. I have it planned for the future though.

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

***JKortesi81*** · 03-29-2019, 06:53 PM

Pay that man his money

**esilverm** · 03-30-2019, 10:42 AM

EDIT: If you are seeing this now don't worry. I had a problem with the saving of users which crashed my bot and deleted all the saved users. I found a copy of it on my computer so we are all good

Pass Forfeit (D) ● Player Profile
Kraken

Special thanks to Blitz, jhockey, Ragnarr, and sulo for the gorgeous sigs <3

[Image: DG0jZcS.png]

academydropout · 03-30-2019, 11:52 AM

03-29-2019, 12:35 PM.bojo Wrote: This is great. If you want to use data from the bojo box I can write some api

:eyes:

Navigation

Extra Menu

About us