Node Js Auto Restart All Forever Js Process When Server Goes Down / Crashes
I am using forever js to keep my node server running 24/7 on AWS EC2. I use this command forever start index.js However, I notice that some time it randomly stops the process and
Solution 1:
A (NodeJS) server should not stop for no reason. Most of the time, it's because of a 500 Error that have not been catched and stop the server, then you will have to restart it.
forever is using node by default to start your server.
nodemon is a npm package that restart your server when the code changes or when your server stops.
You can use forever and nodemon together by doing :
forever start nodemon --exitcrash app.jsor
forever start -c nodemon app.js
Or, as suggested in other answers, you can use PM2, which would be better for production !
Solution 2:
I'd suggest the usage of PM2
This is the best option to run on a production server.
What are the advantages of running your application this way?
- It's easy to setup and run.
- PM2 will automatically restart your application if it crashes.
- PM2 will keep a log of your unhandled exceptions - in this case, in a file at
/home/safeuser/.pm2/logs/app-err.log. - With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically meaning your node application will start as a service.
Solution 3:
So this is an example of using cronto run scripts that can restart service/perform some automated task. Basically, I created some scripts that I need to run at certain time intervals on my server. For your case, you want to make a script that will automatically check the state of your forever.js and if it returns a bad response, run the forever restartall command that you mention above.
You can set this up by creating a new crontab entry on the server. As far as the script goes, I'm by no means a bash script guru; I made a simple script that works for me. Here is an example of checking a service on my machine, restarting it if it is not running.
#!/bin/bash
zabbix_server="service zabbix-server"
zabbix_agent="service zabbix-agent"
logfile=zabbix_auto_restart.log
logfilePath=/etc/scripts/zabbix/$logfile
zabbix_server_running=0
zabbix_agent_running=0
grep_agent (){
local retval=$(ps -ef | grep -v grep | grep zabbix_agentd | wc -l)
echo$retval
}
grep_server (){
local retval=$(ps -ef | grep -v grep | grep zabbix_server | wc -l)
echo$retval
}
check_zabbix_agentd (){
if (( $(grep_agent) <= 0 ))
then
sudo /etc/init.d/zabbix-agent start
echo `date` "$zabbix_agent was stopped... Restarting" >> $logfilePathecho"************************************************" >> $logfilePath#Send email to notify that the script ranecho"$(date)$zabbix_agent was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" <my-email>
elselet zabbix_agent_running=1
fi
}
check_zabbix_server (){
if (( $(grep_server) <= 0 ))
then
sudo /etc/init.d/zabbix-server start
echo `date` "$zabbix_server was stopped... Restarting" >> $logfilePathecho"************************************************" >> $logfilePath#Send email to notify that the script ranecho"$(date)$zabbix_server was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" evan.bechtol@ericsson.com
elselet zabbix_server_running=1
fi
}
main_loop (){
until ((zabbix_server_running == 1 && zabbix_agent_running == 1));
do
check_zabbix_agentd
check_zabbix_server
sleep 1.5
done
}
main_loop
Solution 4:
This video is part of an awesome node-deployment-series ending up with using a service on a UNIX-server... Maybe it helps you. By the way, it's worth watching all four videos... Youtube-Link
Solution 5:
Have you ever considered pm2 as an alternative. pm2 has some pretty neat features like:
- Running a cluster
- Restarting instances in the cluster one by one (zero downtime deployments)
- Resource monitoring (shows realtime cpu and memory usage)
- Manage all pm2 processes using the cli
You can control restarts also by adding a --restart-delay <delay> option.
Hereby a full output of the help of this tool.
pm2 --help-------------
Looking for a complete monitoring and management tool for PM2?
_ _ _ _
|| _____ _ _ _ __ ___ ___||_ _ __(_) ___ ___ (_) ___
||// _ \ |||'_ ` _ \ / _ \ __| '__||/ __/ __|||/ _ \
|< __/|_||||||| __/|_|||| (__\__ \_|| (_) ||_|\_\___|\__, |_||_||_|\___|\__|_||_|\___|___(_)_|\___/|___/
Features
-RealTime Dashboard
- CPU/Memory monitoring
- HTTP monitoring
- Event notification
- Custom value monitoring
-RealTime log display
Checkout
https://keymetrics.io/-------------
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
Usage: pm2 [cmd] app
Commands:
start [options] <file|json|stdin|app_name|pm_id...>startand daemonize an app
deploy <file|environment> deploy your json
startOrRestart <json>startor restart JSON file
startOrReload <json>startor gracefully reload JSON file
startOrGracefulReload <json>startor gracefully reload JSON file
stop [options] <id|name|all|json|stdin...> stop a process (tostart it again, do pm2 restart <app>)
restart [options] <id|name|all|json|stdin...> restart a process
scale <app_name><number> scale up/down a process in cluster mode depending on total_number param
reload <name|all> reload processes (note that its for app using HTTP/HTTPS)
gracefulReload <name|all> gracefully reload a process. Send a "shutdown" message tocloseall connections.
id <name>get process id by name
delete<name|id|script|all|json|stdin...> stop anddelete a process from pm2 process list
sendSignal <signal><pm2_id|name> send a system signal to the target process
ping ping pm2 daemon - if not up it will launch it
updatePM2 updatein-memory PM2 withlocal PM2
update (alias) updatein-memory PM2 withlocal PM2
install|module:install <module|git:/> install orupdate a moduleand run it forever
module:update<module|git:/>update a moduleand run it forever
module:generate [app_name] Generate a sample moduleincurrent folder
uninstall|module:uninstall <module> stop and uninstall a module
publish|module:publish Publish the module you are currently onset<key><value> sets the specified config <key><value>multiset<value>multiset eg "key1 val1 key2 val2
get [key] get value for <key>
conf [key] [value] get / set module config values
config <key> [value] get / set module config values
unset <key> clears the specified config <key>
interact [options] [secret_key|command] [public_key] [machine_name] linking action to keymetrics.io - command can be stop|info|delete|restart
link [options] [secret_key|command] [public_key] [machine_name] linking action to keymetrics.io - command can be stop|info|delete|restart
web launch a health API on port 9615
dump dump all processes for resurrecting them later
save (alias) dump all processes for resurrecting them later
resurrect resurrect previously dumped processes
startup [platform] auto resurrect process at startup. [platform] = ubuntu, centos, redhat, gentoo, systemd, darwin, amazon
logrotate copy default logrotate configuration
generate generate an ecosystem.json configuration file
ecosystem generate an ecosystem.json configuration file
reset <name|id|all> reset counters for process
describe <id> describe all parameters of a process id
desc <id> (alias) describe all parameters of a process id
info <id> (alias) describe all parameters of a process id
show <id> (alias) describe all parameters of a process id
list list all processes
ls (alias) list all processes
l (alias) list all processes
status (alias) list all processes
jlist list all processes in JSON format
prettylist print json in a prettified JSON
monit launch termcaps monitoring
m (alias) launch termcaps monitoring
flush flush logs
reloadLogs reload all logs
logs [options] [id|name] stream logs file. Default stream all logs
kill kill daemon
pull <name> [commit_id] updates repository for a given app
forward <name> updates repository to the next commit for a given app
backward <name> downgrades repository to the previous commit for a given app
gc force PM2 to trigger garbage collection
deepUpdate performs a deep update of PM2
*
Options:
-h, --help output usage information
-V, --version output the version number
-v --version get version
-s --silent hide all messages
-m --mini-list display a compacted list without formatting
-f --force force actions
-n --name <name> set a <name> for script
-i --instances <number> launch [number] instances (for networked app)(load balanced)
-l --log [path] specify entire log file (error and out are both included)
-o --output <path> specify out log file
-e --error <path> specify error log file
-p --pid <pid> specify pid file
-k --kill-timeout <delay> delay before sending final SIGKILL signal to process
--max-memory-restart <memory> specify max memory amount used to autorestart (in megaoctets)
--restart-delay <delay> specify a delay between restarts (in milliseconds)
--env <environment_name> specify environment to get specific env variables (for JSON declaration)
-x --execute-command execute a program using fork system
-u --user <username> define user when generating startup script
--hp <home path> define home path when generating startup script
-c --cron <cron_pattern> restart a running process based on a cron pattern
-w --write write configuration in local folder
--interpreter <interpreter> the interpreter pm2 should use for executing app (bash, python...)
--interpreter-args <arguments> interpret options (alias of --node-args)
--log-date-format <momentjs format> add custom prefix timestamp to logs
--no-daemon run pm2 daemon in the foreground if it doesn't exist already
--skip-env do not refresh environmnent on restart/reload
--source-map-support force source map support
--only <application-name> with json declaration, allow to only act on one application
--disable-source-map-support force source map support
--merge-logs merge logs from different instances but keep error and out separated
--watch [paths] watch application folder for changes
--ignore-watch <folders|files> folder/files to be ignored watching, chould be a specific name or regex - e.g. --ignore-watch="test node_modules "some scripts""
--node-args <node_args> space delimited arguments to pass to node in cluster mode - e.g. --node-args="--debug=7001 --trace-deprecation"--no-color skip colors--no-vizion start an app without vizion feature (versioning control)--no-autorestart start an app without automatic restart--no-treekill Only kill the main process, not detached children--no-pmx start an app without pmx--no-automation start an app without pmx
Basic Examples:
Start an app usingall CPUs available +set a name :
$ pm2 start app.js -i 0--name "api"
Restart the previous app launched, by name :
$ pm2 restart api
Stop the app :
$ pm2 stop api
Restart the app that is stopped :
$ pm2 restart api
Remove the app from the process list :
$ pm2 delete api
Kill daemon pm2 :
$ pm2 kill
Update pm2 :
$ npm install pm2@latest-g ; pm2 update
More examples in https://github.com/Unitech/pm2#usagefeatures
Deployment help:
$ pm2 deploy help
Post a Comment for "Node Js Auto Restart All Forever Js Process When Server Goes Down / Crashes"