Giantbomb Pi Downloader

Morning All,

I've just finished putting the final touches on my new app and would love if a couple of people could test incase I've missed something :)

Over the last 12months I've developed the Giantbomb Pi Downloader. A php application you host on your Pi which does the following:

• Checks for New Giantbomb Videos every Hour
• Download videos onto your Pi, ready to download from Pi saving you time
• My Favourite Feature!!! Ability to set up automatic downloads of your favourite video series (rules page)
• Integration with Slack to sign up to notifications when new videos can be downloaded or have been downloaded

And looks like this:

I still have some loose ends to tie up and a couple of features I'd like to add but it's ready for the world to use or at least it should be....

You can find the project over at Github including a installation script here - https://github.com/Adam2Marsh/giantbomb_downloader

If you have a Pi and can give it a go that would be awesome, just want to iron out any issues I've missed before posting in the main area for everyone to use.

Thanks Guys!

I'll probably give this a go at the weekend. Looks intriguing!

Cool, I will probably check this out this week. I'm guessing this will work on any Debian running machine? Or do you have certain checks for Pi?

Technically it should work with most Ubuntu/Debian based machines but I've only tested on Raspberry Pi running Jessie.

I also forgot add (will update github) but as the videos are downloaded and saved on Pi if you install a DNLA Server and do a little config you can serve the GB videos locally to any DNLA clients like a smart tv :)

Hi

First of all, great work!

- I am running Ubuntu server 16.04. I when through your bash script manually doing each step individually.

- php 7.1 isn't in Ubuntu repos. but went through apt-get installing each package individually getting php7.0 versions without issue

- Had a problem running

php /opt/giantbomb_downloader/artisan migrate --force

which gave the error:

PHP Warning: require(/opt/giantbomb_downloader/bootstrap/../vendor/autoload.php): failed to open stream: No such file or directory in /opt/giantbomb_downloader/boo tstrap/autoload.php on line 17

PHP Fatal error: require(): Failed opening required '/opt/giantbomb_downloader/bo otstrap/../vendor/autoload.php' (include_path='.:/usr/share/php') in /opt/giantbom b_downloader/bootstrap/autoload.php on line 17

- After some googling running seemed run through huge amount of installing etc (too long to log here) then it worked:

sudo composer update --no-scripts

- Running composer.phar without root got the following with permission error?

./composer.phar install

Cannot create cache directory /home/nas/.composer/cache/repo/https---packagist.org/, or directory is not writable. Proceeding without cache

Cannot create cache directory /home/nas/.composer/cache/files/, or directory is not writable. Proceeding without cache

Loading composer repositories with package information

Installing dependencies (including require-dev) from lock file

Nothing to install or update

[ErrorException]

file_put_contents(/opt/giantbomb_downloader/vendor/composer/installed.json): failed to

open stream: Permission denied

- Running with sudo seemed to work but with a warning:

Do not run Composer as root/super user! See https://getcomposer.org/root for details

Loading composer repositories with package information

Installing dependencies (including require-dev) from lock file

Nothing to install or update

Generating autoload files

ocramius/package-versions: Generating version class...

ocramius/package-versions: ...done generating version class

> php artisan clear-compiled

The compiled class file has been removed.

> php artisan optimize

Generating optimized class loader

Compiling common classes

- My earliest video listed is the following. Does this use RSS to get video list or any way of going back further? Had feeling this is limited on the Giantbomb side?

- I was having an unhelpfully vague [Object] [object] message/error from chrome when loading the home page. Redoing chown + chmod of opt/gbdownloaded directory seemed to get rid of this

- You mentioned features, if possible a option to specify where videos are downloaded. I have my OS on a USB stick attached to a NAS. Got around this by making a sym link to my storage from opt/gb/storage etc folder.

- I do have a couple of small issues: My videos are all duplicates ie. each video is listed twice/duplicated. When I was troubleshooting I ran some of the commands more than once, such as the following, could that have caused the issue? Done some exploring but can't seem to find where this info stored or can be reset.

sudo composer update --no-scripts

sudo php /opt/giantbomb_downloader/artisan schedule:run >> /dev/null 2>&1

and then:

sudo php /opt/giantbomb_downloader/artisan schedule:run

- When click "Save" on video it will download but the bar above doesn't change from 0.00B. I am not sure ifs something to do with not loading image assets full stop, as the splash page had a empty box indicating an image not loaded.

- If you are on page 2 and click download it will do this but knocked you back to the first page each time.

- How do you use the rule page? I can add a "Video name" text string and click enable and it will add it self to current rules. For example, can I add quick looks?

Again, massive thumbs up. This will be extremely useful. Above just some queries and hope useful testing info - not a moan or nitpicking I hope. Sorry, my programming knowledge is very limited!

I had the same experience as the above poster but on Debian Jessie x64. Had to install apache2 manually, the custom repo wasn't working, php7.1+ not in Debian repos either.

I know this tool is for pi, so I'm not complaining, just find it a little bit sad, since it "should" work.

Hi Guys! Thanks for giving it a go for me, I expect there to be a couple of little issues and some areas for me to improve :) The feedback so far it perfect.

@wilsouk: I hadn't tested on Ubuntu but should be able to update the automated install to make it easier next time. Hopefully I've captured everything, if I've missed anything let me know.

Before you can run any of the php artisan you have to run

./composer install

Inside /opt/giantbomb_downloader as this grabs all the dependent packages for the site to run, it seems alot but its just how the framework I made the app on works. Once you have that done all the "php artisan" commands should work :) Running the composer command is not recommended with Sudo because the packages then have sudo permissions, if you chown the folder and all sub folders to your user you should then be able to run composer without sudo.

Currently the app grabs the last 30 days of videos, I can improve this to give you an option to go back further. How far would you want to go back? Was you hoping to be able to see everything or just go back like 90 days?

The Object Object error is annoying as I saw it once, the chown + chmod of opt/gbdownloaded will fix the issue. I'll add onto my BUGS list to look at fixing.

The storage location of videos in on my TODO list, I'll push it up the chain and get that configuration out ASAP but the symlink will work fine (nice idea!).

Hmmm the duplicated videos is odd... that shouldn't happen as there are checks to make sure it doesn't store the information of a video twice.... to start again afresh do the following commands:

cd /opt/giantbomb_downloader

php artisan migrate:refresh

Have you changed the location of the app from "/opt/giantbomb_downloader" as there is a cron job which will fail also the supervisor config "laravel-worker.conf" which are hard set in the scripts/configs which you'll need to change. The cron job handles checking for new videos every hour, the supervisor config is what handles the actual downloading of any videos you've clicked save on. The video you clicked save on does it show a status of queued?

I will look to make it so if you click save on a different page it takes you back to the same page, adding to BUGS list.

So the rules page uses a regex check on what you enter, so if you enter "Quick Look" any videos which have that inside it will auto download just like you said. I'll add some extra explanation to how the Rules section works, I was thinking about changing it to Subscription maybe? As rules isn't the best name.

Thanks for giving it a try, the feedback you provided is perfect :) It's more than what I was hoping for.

@rocketblast0063: Thanks for giving it ago, originally I was thinking most people who might use would have a Pi. I have under estimated what people have inside there house :) I should be able to pretty easily make the automated script work for Ubuntu, have added to my list of TODO :)

If you guys (or anyone else) are on the Discord GB channel and have some issues. Then give me a shout on there, i'm Adam2Marsh.

I have also got a Slack, more than happy to create a channel and send some invites if that works any better for people.

@wilsouk If the downloadfs are not working can you run the following:

sudo supervisorctl status

You should get an output of:

laravel-worker:laravel-worker_00 RUNNING pid 775, uptime 8 days, 20:26:09

If not then something didn't get installed correctly as that process is what handles the downloading.

Seems like a fun little thing to try out.

Cheers for the reply.

Since my last post, and without me doing anything, the latest video (VR rodeo) has appeared and isn't a duplicate so its working fine.

I tried sudo supervisorctl status and that gives the proper message.

I ran php artisan migrate:refresh and initially got error that application in process. Running sudo supervisorctl stop all then running php artisan migrate:refresh did rest the downloader (it asked me to put in API key again, refreshed my videos list, reset rules to blank and which videos marked as download - which I was expecting!). But strangely all the videos again were duplicates, even more strangely, the most recent video (VR rodeo) was not. Is it calling on same local database or something again? Then again this not issue for me as working great now - only mention it if bug for you.

I have kept app in opt/giantbomb_downloader (as default) so think cron job working as mentioned it had refresh to have VR rodeo on there.

I have added some rules and will let you know how I get on.

Regarding how many days - I feel 30 days fine, would more slow down? Just wonder whether would useful to download series i.e. Yakuza from start but TBH fine at 30 days for me is attraction is downloading videos ongoing.

@wilsouk: Really odd with the duplicate videos, could you send me the log file? it's inside "/opt/giantbomb_downloader/storage/logs/". That should help give me a better idea what's going on :) as am confused currently.

So when starting from scratch again it should use the gb api and get the latest 30 videos... it you want to force a refresh the best way currently is change the url from "/videos" to "/NewVideos" it should give you a nice plaintext view of the last 30 videos and advise what is new and what is not. It's what the cron job calls just internally instead of from a http call.

Let me know how you get on with the rules, they currently only work on new videos added. Doesn't backdate but have added it onto "To Think" list.

I do have a card around being able to search the GB Video list and add videos locally, awesome idea to include grabbing a while series.

I've made my trello board public so if you want to check out what I've added go ahead: https://trello.com/b/l4yppMfy/gb-downloads-site

It is a little messy, plan to tidy up tomorrow.

Thanks again to everyone trying it out, the feedback has been great.

I can recommend having a look at the project PiHole https://pi-hole.net/ for reference. This for example: https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh

It's a script and tool made for Pi but works out of the box for Debian (and Ubuntu) too. I'm not a programmer or developer so I can't say how hard or how much work is needed, but it seems to be a rather easy fix due to Raspberry already running Debian.

Anyways, interesting and useful tool, thanks for making it!

@rocketblast0063: Thanks for the link, I actually based my install script off there's :) Just removed some of the more complex parts but will spend some time to understand it a little better :) Not yet configured but I was going to add some extra config if you have pi-hole installed so you could reach site via "gb.pi" instead of the ip. It's what I do on my home setup just haven't wrote the docs or setup script for it yet.

Thanks for feedback, super happy people are using it :) First time releasing a tool into the wild and so far looks good :D

@wilsouk: Hi Wilsouk. Just testing on my personal install the ability to change default download directory, going to run for week but if all is good will push to master next Monday :) Also includes the ability to install via GUI and a few other fixes.

If anyone is having any issues let me know, still got a few more things to do and tidy some backend up. Am off on holiday next week so I might be a little slow in responding but will keep an eye out.

Hi All,

Happy to announce a new version of the app has been released, new features are:

• When you run for the first time it will load some videos automatically
• There is now a refresh button on main screen to check for any new videos
• New Config so you can save your videos somewhere else than the default like a connected harddrive :) (Just make sure it's writeable) @wilsouk

I have also started a new update method, this isn't 100% yet and not working as well but you'll see it there :)

How to update

- Log onto server
- cd into giantbomb directory (default is /opt/giantbomb_downloader) "cd /opt/giantbomb_downloader"
- grab latest code by running "git pull"
- Restart Supervisor process for new changes "sudo supervisorctl restart laravel-worker:*"

Got this ready just in time for E3 so you can now connect a hardrive and use the rules to grab every video at ease. Some of the directory sizing isn't working aswel so if you do use an external drive just make sure you don't run out of room :)