[mysqld]
log-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式
server_id=1 #配置mysql replaction需要定义,不能和canal的slaveId重复
canal的原理是模拟自己为mysql slave,所以这里一定需要做为mysql slave的相关权限
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' IDENTIFIED BY 'canal' WITH GRANT OPTION;
FLUSH PRIVILEGES;
针对已有的账户可通过grants查询权限:
show grants for 'canal'
启动步骤:
1. 下载canal
下载部署包
wget http://canal4mysql.googlecode.com/files/canal.deployer-1.0.0.tar.gz
or
自己编译
git clone git@github.com:otter-projects/canal.git
cd canal;
mvn clean install -Dmaven.test.skip -Denv=release
编译完成后,会在根目录下产生target/canal.deployer-$version.tar.gz
2. 解压缩
mkdir /tmp/canal
tar zxvf canal.deployer-1.0.0.tar.gz -C /tmp/canal
解压完成后,进入/tmp/canal目录,可以看到如下结构:
drwxr-xr-x 2 jianghang jianghang 136 2013-02-05 21:51 bin
drwxr-xr-x 4 jianghang jianghang 160 2013-02-05 21:51 conf
drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib
drwxr-xr-x 2 jianghang jianghang 48 2013-02-05 21:29 logs
3. 配置修改
公用参数:
vi conf/canal.properties
#################################################
######### common argument #############
#################################################
canal.id= 1
canal.address=
canal.port= 11111
canal.zkServers=
# flush data to zk
canal.zookeeper.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 32768
## detecing config
canal.instance.detecting.enable = false
canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false
# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transactionn.size = 1024
# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30
#################################################
######### destinations #############
#################################################
canal.destinations= example
canal.instance.global.mode = spring
canal.instance.global.lazy = true ##修改为false,代表立马启动
#canal.instance.global.manager.address = 127.0.0.1:1099
canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml
应用参数:
vi conf/example/instance.properties
Instance.properties
#################################################
## mysql serverId
canal.instance.mysql.slaveId = 1234
# position info
canal.instance.master.address = 127.0.0.1:3306 #改成自己的数据库地址
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
# username/password
canal.instance.dbUsername = retl #改成自己的数据库信息
canal.instance.dbPassword = retl #改成自己的数据库信息
canal.instance.defaultDatabaseName = #改成自己的数据库信息
canal.instance.connectionCharsetNumber = 33 #改成自己的数据库信息
canal.instance.connectionCharset = UTF-8 #改成自己的数据库信息
# table regex
canal.instance.filter.regex = .*\\..*
#################################################
说明:
canal.instance.connectionCharset 代表数据库的编码方式对应到java中的编码类型,比如UTF-8,GBK , ISO-8859-1
canal.instance.connectionCharsetNumber 代表数据库的编码方式对应mysql中的唯一id,详细的映射关系可查看:com.mysql.jdbc.CharsetMapping.INDEX_TO_CHARSET
针对常见的编码:
utf-8 <=> 33
gb2312 <=> 24
gbk <=> 28
4. 准备启动
sh bin/startup.sh
5. 查看日志
vi logs/canal/canal.log
2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
具体instance的日志:
vi logs/example/example.log
2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
6. 关闭
sh bin/stop.sh
使用中出现以下错误:Could not find first log file name in binary log index file
2015-01-15 10:40:01.756 [destination = example , address = /127.0.0.1:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:example[java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:95)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:116)
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:209)
at java.lang.Thread.run(Thread.java:662)
最终解决因为没有权限,使用root用户运行成功!
相关推荐
mysql主从同步及canal配置详解.docx
配置详见:https://github.com/alibaba/canal/wiki/AdminGuide同步实例配置6 # enable gtid use t
Canal环境配置信息文档,主要是描述Canal怎么配置,和监听多数据库的时候该怎么配置。资料自己整理的
阿里的开源canal通过订阅binlog可以生产mq消息,实现了数据链路的对接和输出,为我们在做业务兼容,业务迁移的过程中提供了很多便利,但是个人在工作中发现有一个问题,就是目前个版本生产的消息都没有key和tag,...
Linux下Canal安装和使用
mysql需开启binlog 查看是否开启binlog SHOW VARIABLES LIKE '%log_bin%' 如果log_bin的值为OFF是未开启,为ON是已开启...canal下载及配置 https://github.com/alibaba/canal/releases/tag/canal-1.1.5 配置文件见附件
canal是阿里巴巴旗下的一款开源项目,纯Java开发。基于数据库增量日志解析,提供增量数据订阅&消费,目前主要支持了MySQL(也支持mariaDB),支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x.
在springboot中整合与使用canal
jdk17安装Canal的安装文件,解压缩之后修改配置文件即可
canal-canal-1.0.22_源码 canal-canal-1.0.22_源码 canal-canal-1.0.22_源码
canal配置文件
为 php 开发者提供一个更友好的使用 Canal 的方式。Canal 是mysql数据库binlog的增量订阅&消费组件。 基于日志增量订阅&消费支持的业务: 数据库镜像 数据库实时备份 多级索引 (卖家和买家各自分库索引) search ...
最新版阿里开源中间件canal实现mysql数据库同步,零侵入不写代码实现,也可以通过整合到项目程序实现更加灵活的控制。详细使用方法:https://blog.csdn.net/u014374009/category_9409106.html
mysql+canal+kafka配置及python简单实现!
canal.adapter-1.1.4.tar.gz canal.admin-1.1.4.tar.gz canal.deployer-1.1.4.tar.gz 文件比较大,分两部分分上传
canal.adapter-1.1.4.tar.gz canal.admin-1.1.4.tar.gz canal.deployer-1.1.4.tar.gz 文件比较大,分两部分分上传
canal客户端-canal.deployer-1.1.7-SNAPSHOT.tar.gz
包含canal全套资源包含以下 canal.adapter-1.1.5.tar.gz canal.admin-1.1.5.tar.gz canal.deployer-1.1.5.tar.gz canal.example-1.1.5.tar.gz